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PREFACE - AN OVERVIEW OF THIS REPORT 



This report is addressed to the problems of how s^Jstematically to ob- 
tain student opinion about the teaching to which such students are exposed 
in a University setting. Nowhere do we consider the very important matter 
of hcyw to interpret such "uneducated" opinion** We call such opinion "student 
evaluation of teaching" without any implied judgment concerning the validity 
of this student evaluation. 

This report specifically describes the procedures presently used by 
the Department of Physics and Astronomy. (These procedures have been given 
the acronym FATS - for Physics and Astronomy Teaching Survey.) In order to 
develop PATS, many general problems had to be addressed and decisions made. 
In particular it was found that "purely technical" questions were inescap- 
ably intertwined with "philosophical" questions » and many such philosophical 
questions had to be asked and answered before one could fully implement PATC* 
Such questions must always arise in any atteiq)t to obtain student evaluation 
of teaching. The danger is that certain administrative type decisions on 
"technical questions" may be made rithout full realization that they have 
philosophical import. 

It follows that this report must address itself both to general ques- 
tions relating to all such student evaluation procedures, and to the specific 
procedures used in PATS. We hence decided to write this report in two sepa- 
rate parts, separately bound. Part I deals primarily with general questions, 
and should be of Interest to anyone who is attempting to set up a student 
evaluation procedure. Part II is-primarily an operating manual for PATS in 
its present form. It would be iyf interest to someone who is charged with 
the actual implementation of such a similar procedure, since he must deal 
with all the technical questions. But our hope is that all general type 
questions are raised and discussed in Part I. These are the types of ques- 
tion which would need to be considered, for instance, by a faculty committee 
overseeing the implementation of such a procedure. 

The two parts of this report are so written that each may be read on 
its own. Hence it has been necessary to duplicate certain material. Struc- 
turally Part II is the continuation of Part I, and the two parts together 
form the conplete report. 

We have organized the report as follows: Part I consists of Chapter 1 
and 2 and related appendices. Chapter 1 provides a general overview of 
PATS, from the nature of the questionnaiires to a brief description of the 
way in which the responses are processed and the results made public. A 
sunmary of the entire process is given in the final section of this chapter. 
Chapter 2 considers some of the problems and decisions involved in such a 
project, such as how one goes about combining the results of different 



But is it any more uneducated then most other forms of teaching evalua- 
tion? 



sections. This chapter explains how these questions were resolved for PAIS, 
some of them being decisions of taste or convenience, with little compelling 
force behind them. Since most of the "philosophical" problems are connected 
with combining the results of different sections, this chapter is principally 
a discussion of procedures used it* the conqputer program AGGFORiM.** A number 
of concepts and definitions are introduced in this chapter. 

Part IT of the report consists basically of "how to" manuals for the 
actuaJ. execution of the various aspects of PATS. Chapter 3 discusses over- 
all administration and supervision. Chapter 4 treats the preparation of 
data for and execution of DATAREAD and INITPRT. Finally, Chapter 5 simi- 
larly discusses the preparation of data for, and execution of, AGGFORM. 
(There are a"^so numerous appendices.) We have atteTq)ted to make these three 
chapters and r^'lated appendices each complete enough in itself so that, for 
example, different individuals could be responsible for different phases of 
the survey, and each would only have to read the one chapter relevant to his 
phase. Since various concepts and definitions are introduced in thjs report 
as needed, and a particular chapter may assume that the reader has axready 
learned a particular term, such definitions have been collected in a Glossary 
for the convenience of those using, say just one chapter of the leport. In 
some cases the Glossary contains illustrative examples not found in the main 
text. This Glossary is attached to the report as an Appendix which 
appears in Part II. 



While PATS is an "official" survey procedure as presently implemented 
by the Department of Physics and Astronomy, it must be emphasized that this 
report is the work of four individuals; they were all deeply involved in 
setting up PATS, and hence they (particularly Dr. Clauue Kacser) were forced 
to make many "philosophical" decisions as they arose during the task of tech- 
nical development. This report discusses these decisions. It follows that 
all such discussion should definitely not be construed as representing the 
"view of the Department of Physics and Astronomy," but solely as representing 
the views of the four individual authors. 



PATS is an ongoing operation. Hence it is continually being modified, 
updated, and hopefully improved. At any given stage, some minor errors 
exist. Xt has not seemed worthwhile to attempt to correct all of them in 
this present report. Thus questionnaire 3 contains some unfortunate "typo- 
graphical" errors, which will be corrected in the next run. But such 
errors hopefully should not detract from the overall value of this report. 
For similar reasons, we include unedited internal department memos. 



We consistently use the term "section" to denote the smallest basic unit 
of teaching exposure to one instructor, whether this refers to lecture, recita- 
tion or lab; faculty or graduate T/A; and whether one section of a many section 
course, or one distinct course. 

DATAREAD, INITPRT, and AGGFORM are the names of three separate electronic 
computer programs that are used by PATS at various stages. 
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CHAPTER 1^ A COMPLETE GENERAL DESCRIPTION OF PATS 
1.1 Introduction - Why Such a Report at All? 

"Why is this report so huge? Vlhat is so special or significant about 
a teadiiug survey questionnaire? Once one ha,s decided to ask students to 
evaluate teaching, surely it is straightforw.ird to set up and administer 
the instruiients needed?" It is likely th^t such questions, or similar ones, 
will already have occurred to the reader. An.i uiiO can understand and sym- 
pathize with such an initial response* 

Yet in fact as one proceeds into the de/elopment of the necessary 
"machinery," one discovers that there are many "philosophical" questions 
that must be decided, and that the answer to such questions lead to quite 
complicated administrative type decisions and procedures* Furthermore, 
and most important, if one ignores initial philosophy and instead procedes 
pragmatically and simply devises a "straightforward" overall procedure, it 
may turn out that the procedure undermines the utility of the overall ques— 
tionnaire; or worse still, the procedure may yield "results" which are in fact 
misleading, and hence dangerous in the extreme* This definitely should 
not be allowed to occur In an official departmentally sponsored teaching 
survey (nor anywhere else if possible)* 

We believe that many featur^:s of the detailed procedure described in 
this report go a very long way towards establishing the utility and "relia- 
bility" of such a teaching survey,, of fe ring definite improveraents over previous 
such attempts of which we are aware* Our overall viewpoint has been that if 
something is worth doing, it is worth doing well* (One might argue that 
half a loaf of bread is better than none, but there is great danger in such 
an approach*) 

The greatest difficulty in the use of such questionnaires occurs when 
one attempts to compare one instructor with another, or with an overall 
"average instructor," or perhaps to place a particular instructor at a cer- 
tain percentile in an overall instructor distribution* And yet in actuality 
all such survey results must necessarily be used primarily In a con^arative 
way* 

Why is conparative use difficult? People are aware in theory of the 
impossibility of comparing apples and pears; yet the basic error that many 
questionnaire teaching evaluation procedures inescapably fall into is the 
attempt to compare one apple with a whole bowl of fruit - apples, pears, 
grapefruit, etc*, with perhaps a few vegetables also thrown in at random 
for good measure « 

i 

Consider first all possible University courses and instructors* Clearly 
one should not attempt to compare instructors of le'*;ure style courses, with 
1 instructors of discussion or recitation classes: nor (presur.uibly) should one 

^ compare faculty members with graduate teaching assistants* But this still 

i leaves a very large number of faculty-* taught lecture course experiences with- 

f in a university* Can these all be compared one with another* or a specific 

I one with some sort of "average"? We assert vehemently "NO"^ For example, 

I lecturers In different disciplines will often need quite different skills* 
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The present report does not however delve into the question at this leval, 
since it is addressed primarily to courses offered by the Department of 
Physics and Astronomy, 

Yec even if one considers only physics lecture courses taught by 
faculty, and only attetnpns to evaluate thv2 quality of the lecture as judged 
by the students , we still are left with not only "apples" anu "pears," but 
the whole basket of fruit. Consider the difference between: 

a) large enrolment freshmen or sophomore courses, possibly meeting 
General Education requirements, where the student is a s»:ml-willing captive, 
but without a deepseated interest in the subject; 

b) large enrolment freshmen or sophomore "service" courses forming a 
specific requirement for some other Department's major, e.g. required for 
Engineers or for Pre-Meds; 

c) small enrolment lecture sections (20-30 students) of courses aimed 
I physics majors. 

Typical students in sucli different courses will have different expec- 
tations, requirements and bases of coiq>arison, when attempting to evaluate 
these courses. Consider as an extreme example the "Course Guide" question 
"I would recommend this course to anyone." The danger in this particular 
example is rather obvious if one thinks about It, but many other apparently 
more absolute questions will also be answered veiy differently by such dif- 
ferent students. Furthermore, the actual goals of such different classes 
of courses are different, and hence different questions should be asked 
for these different types of courses. 

In the present survey, the three different faculty lecture experiences 
(a), (b), (c) above are surveyed using three differ ent questionnaires. In 
fact, to survey all physics learning experiences, we distinguish between nine 
different teaching experiences, using appropriate questionnaires. These 
different questionnaires ask different questions (though many questions do 
appear on more than one questionnaire). This device automatically prevents 
too easy apparent comparisons between different teaching experiences. But 
this device only serves to separate the "fruits" from the "vegetables." 

Even, for instance, within the class of small physics major lectures we 
still have "apples" and "pears." For example, consider the differences be- 
tween a specialist graduate course, an introductory graduate course, a senior 
undergraduate elective, and the initial freshman course; or between a deman- 
ding mathematical techniques course, and an interdisciplinary survey course 
such as biophysics. The present survey PATS deals with this problem by use 
of the concept of "a heirarchy of aggregates." We believe that this is a 
very important and powerful concept in such teaching evaluation student sur- 
veys. A very considerable part of the complexity of PATS arises due to 
implementation of this concept. To the best of our knowledge such a concept 
has not been implemented in any similar student teaching evaluation sun/ey. 



It forms a crucial aspect of PATS. 



What precisely is this concept? It simply means that for any given sec- 
tion we form not one tuit many "norms," at different levels, with which to com- 
pare that section. Let us explain further: within the class of physics ma- 
jor lectures, there are many different individual sections, each (in general) 
taught by a different faculty member. Hie same questionnaire is administered 
to each such section. Within a given single section, the individual student 
responses are first summed for each question. This produces a student eval- 
uation of that section. But this is not very valuable or useful without 
some "norm" to which it may be compared. 

PaTS enables one to obtain a series or heirarchy of such "norms." We 
call these norms "aggregates." An individual aggregate consists of the summed 
section distributions of answers to each question, averaged over all sections 
which seem appropriate for that aggregate. Heirarchies (or sequences) of 
aggregates provide a heirarchy of "average" evaluations with which one can 
compare the evaluation of a particular section of a course. For example 
the heirarchy for a faculty member teaching a first-semester freshman physics 
major course could consist of aggregates for each of the following: 

a) all sections of that course; 

b) all sections of all freshman year physics major courses; 

c) all sections of ^11 first and second year physics major courses; 

d) all sections of all (required) physics major courses. 

Similarly, in the case of a teaching assistant (call him F) who teadies more 
than one section of a particular recitation or lab» a heirarchy might con- 
sist of: 

a) all recitation sections of a particular course (X) taught by T/A F 
under Professor L; 

b) all recitation sections of course X under Professor L; 

c) all recitation sections of course X; 

d) all recitation sections of courses X and Y. 

Such heirarchies of aggregates provide, for each individual section, a 
meaningful set of comparison distributions. As one "goes up the heirarchial 
tree," these comparison groups become more heterogeneous and less meaning- 
ful (though they naturally tend to be less determined by individual varia- 
tions). Notice that a particular "higher" aggregate will appear in many 
different such heirarchies. Even so, the number of such individual aggregates 
is large. For instance, simply for undergraduate physics major courses, 
in Fall 1971, with 26 individual sections, there were 13 aggregates. 

We believe that the computing and printing out of all such heirarchical 
aggregate distributions for a given question within a basically homogeneous 
teaching experience (e.g. all physics major lectures) is probably an essen- 
tial feature of any really useful student teaching evaluation survey. We 
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believe this concept is a significant contribution to the utility and validitv 
of such surveys. To implement It requires a great deal of administrative 
labor. Yet this labor is well worthwhile. 

While the details of such heirarchical sequences of aggregates xvould 
differ from department to department, we believe that the concept is univer- 
sally applicable. Furthermore, it is presumably also applicable across de- 
partments; thus one could imagine a university wide questionnaire aimed at 
all general education courses with various heirarchies of aggregates super- 
imposed, both intra- and inter- departmental. 

We believe that due consideration of such concepts is essential when- 
ever anyone is setting up a student teaching evaluation survey. Thus any 
such person should benefit from a reading of parts of the present report. 
The actual report is a detailed specific description of the procedure "PATS" 
presently used within the Department of Physics and Astronomy. One of its 
purposes is to provide a completely self-contained operating manual for PATS. 
But it has been written with the express intent of also stimulating the de- 
velopment of other such instruments. It is likely that ver>' few people will 
need to, or want to, read this entire report. But . believe that anyone 
who is involved in setting up such a survey will benefit from reading fur- 
ther. 



One other crucial general philosophical remark should be made at this 
time. It is likely that any department or Univ^trsity which sets out syste- 
matically to obtain student evaluation of teaching would be aware of many 
possible overlapping goals, including for instance: 

a) as an aid in. improving teaching; 

b) as an aid in improving courses; 

c) as one facet of evaluating the teaching of faculty and T/As: and 

d) as a diagnostic aid for individual faculty and T/As to use in assis- 
ting themselves in improving the effectiveness of their teaching and of 
their courses. 



It seemed clear at least to some people that if the Department of Physics 
and Astronomy was going to embark on such a student evaluation procedure, it 
should attempt to obtain as much information as possible. Not all of it 
would be germane to a single goal as listed above, but all pieces of infor- 
mation would be useful. 



The alternative approach which uses only very few "subjective" questions 
is probably easier to process. In the view of at least one of the present 
authors (CK) this ease of processing incurs a great loss in utility. We do 
not pursue this discussion further, but wish simply to alert the reader to 
this basic question of approach. 

We now turn from the general to the specific. In the rest of this chap- 
ter we describe in fair detail PATS, the Physics and Astronomy Teaching Survey, 
being the procedure currently used by the Department of Physics and Astronomy 
to obtain student evaluation of all facets of the teaching to which they are 
exposed. 



1.2 Quescionnaire Format 



Many faculty members and T/As used to give out their own questionnaires 
for their own use. Iflien the Department decided to issue an official manda- 
tory questionnaire, the first version (developed under time pressures) was 
a rather brief questionnaire (see Appendix 1, P*A1) which was used in all 
types of courses, for lecture, laboratory and recitation, and for both 
faculty and teaching assistan s. Sample printed-untput for this question- 
naire is given in Appendix 2, p.A2. 

The present format was introduced in Spring 1971. In this format one 
basically attempts to ask ail possible questions, so th-it many different 
uses of the responses can be made. Thus "diagnostic" questions (e.g. "black- 
hoe^d technique") are included since they will be useful to the person 
be^ ; evaluated, even though they are not needed to evaluate the overall 
quality of the teaching. (They are useful in interpreting the evalustions, 
however.) 

Furthermore, different questionnaires were developed for different 
styles of courses or different types of contact. At the present time we 
have seven different questionnaires with differing numbers of questions and 
different questions on them. They are listed in Table 1, and are given in 
fu"* ^ in Appendix 3, p. A4. [As discussed in the preface, these are "working" 
copies as actually issued in Fall 1971.] For purposes of part I rf this report 
only some of the questionnaires are given there. All are given in Pare II. 

Clearly the specific breakdown into questionnaire types will not trans- 
fer CO other departments. On the other hand, the concept of having severa.1 
questionnaires aimed at evaluating different parts of the overall teaching 
mission, and differing for faculty and for T/As, is one that seems important, 
and has later applicability in computing aggregate (normative) responses 
with which individuals may best be compared. (Note that questionnaires 5 and 
6 can each be used either to evaluate the teaching assistant or the faculty 
number in charge - thus we actually have nine different uses of questionnaires.) 

Another feature of the questionnaires is that (in general) each ques- 
tion is followed by a choice of five "spelled-out" responses. For example: - 
course was planned: (a) extremely well; (b) well; (c) acceptably; (d) rather 
poorly; (e) extremely poorly; (f) no answer. Clearly these possible responses 
do form a ranked progression, as do the possible responses to most questions. 
But this "spelling-out" allows for variations such as from "much too easy" 
to "much too hard;" also for simple "yes /no," as well as questions where 
there is no intent to "rank" the responses. [This "speiled-out" format was 
borrowed from the Chemistry Department's questionnaire.] 

Many similar questionnaires use a universal five point scale in one 
form or another; as for instance did the previous short form (Appendix l,p.Al)* 
The "spelled-out" format has the advantages of flexibility, of defining the 
meaning of each response more clearly, and not least of encouraging the 
responding student to think about which response is most appropriate. 

This last point answers the problem of the possible student who decides 
on a universal grade for the course, e.g. B and would make response 2 con- 
sistently on a l-to-5 type questionnaire. With our variable format, response 
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TABLE 1 - Questionnaire Types 

Questionnaire 1 - "General" Courses (Lecturer) - Typically large enrolment 

freshmen or sophomore courses meeting the General Education Requirement. 
(65 questions) 

Questionnaire 2 - "Service" courses (Lecturer) - typically large enrolment 
freshmen or sophomore courses forming a specific requirement for some 
other Department's major. (65 questions) 

Questionnaire 3 - "Physics" Courses (Lecturer) - all lecture courses for 
physics or astronomy majors, undergraduate or graduate (60 questions) 

Questionnaire 4 - Recitations - to evaluate the recitation section (usually 
taught by a T/A) associated with "general" or "service" courses (42 
questions) ^ 

Questionnaire 5 - Structured Laboratory Sections - to evaluate the lab 

section (usually taught by a T/A)associated with "general" or "service" 
courses (20 questions) 

Questionnaire 6 - "uvistructured" lab - to evaluate either the professor or 
the T/A in a separate lab course for advanced students taught in a 
deliberately non-cook-book, non-structured, non-guided manner (pro to- 
real-world-research) (52 questions) 

Q uestionnaire 7 - Laboratory Courses - to evaluate either the professor or 
the T/A in a separate lower level laboratory course. (45 questions) 



A detailed list of the assignment of questionnaires to Physics courses is 
^iven in Part n of this report, where all the questionnaires are also given. 
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(b) does not always correspond to a ''B" grade. Furthermore we deliberately 
"scramble" the order of responses in questions 4 and 8 (e.g. (a) poor, (b) 
good, (c) satisfactory, (d) less than satisfactory, (e) excellent, (f) no 
answer). It is hoped that this conditions the respondent early to read all 
responses carefully. 

Of course one must expect some respondents to answer in a wilful, non- 
serious, random, deliberately vindictive or even deliberately charitable 
or even praising w^. One hopes that the distribution of responses contains 
true information on how the students view the course and instructor. Natur- 
ally they only see one side of the course and may not be able to judge all 
matters fairly. But such caveats do not affect the utility of such ques- 
tionnaires — they simply imply that one must learn to interpret the content 
of the students' responses. 

1.3 Processing the Responses 

Thus far we have considered the individual questionnaires; now we must 
decide what to do with all the responses after we have obtained them. The 
sheer magnitude of the data involved is determined not merely by the large 
number of students enrolled (in Physics and Astronomy courses) but also by 
the number of different questionnaires to which an individual, student may 
respond. For a single course, for example, a student may fill out up to 
three different questionnaires (for the lecture, laboratory, and recitation 
sections of that course), and at least in the case of physics majors, there 
will be questionnaires for more than one course. 

The most obvious first step is simply to summarize the responses for 
each individual section. (The problem of "team-taught" sections, those 
taught by more than one instructor, will be discussed in Chapter 2.) This 
is accomplished in PATS by the computer program DATAREAD, which computes 
the '^distribution of responses" for each section - i.e., for each question, 
hc-w many students chose each of the six possible responses. However, even 
after all the individual questionnaire answers for each individual section 
(lecture, lab, or recitation) are combined there are still several hundred 
sets of results to consider • one set from each section. Not only is this 
an unwieldy amounw of data, but it also fails to allow easy comparisons - 
e.g., the results of one Physics 10 section vs. some sort of average for 
al± Physics 10 sections. It is for these reasons that AGGFORM was written. 

AGGFORH is the computer program that computes and prints out all the 
heirarchies of aggregates discussed in section 1.1. As indicated there, an 
aggregate contains the "sum" of the responses for a particular set of sec- 
tions. (The ''sum" is always left as a distribution of responses to each 
individual question.) Such a "summing" over sections produces what might 
be called an "average" response. But since the term "average" implies, 
among other things, greater statistical significance than is often present, 
we always refer to these sums as "aggregates." 

The choice of which sections should be aggregated, and thus the structure 
of the heirarchies of aggregates, is always left fully to the individual 
running the programs. PATS does follow certain gxiidelines in making these 
choices of heirarchies and these guidelines are discussed later. 
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1.4 Printed Output 

Besides summarizing the results for each section and forming all the 
aggregates desired, the other important function of the computer programs 
is to provide printed results in a readable format. Basically there are 
two different types of printed output - that for single sections, and that 
for aggregates. For single sections the output consists of two distributions 
of responses to each question: l)the number of students who chose each of 
the possible response:; to each question (including "no response"), and 
2) the percentage represented by each such number. Tlius the typical output 
for one section contains 2 x 6 x 50 = 600 numbers (2 distributions, 6 possible 
responses, 50 questions). Sample output on individual sections is given in 
Appendix 4, p.A17. 



The printed output for aggregates is very similar to that for an 
individual section, except that only percentage distributions are provided. 
As explained later, the distributions for aggregates are computed in two 
(or three, if appropriate) different ways; all such distributions appear 
side by side on the printed output* (Again, details on the ways in which 
aggregates are computed will be found in Chapter 2.) Sample aggregate- 
output is displayed in Appendix 4c, p.A21. 

As a policy decision presently in force, we do not compute means or 
standard deviations, i.e. a "score" for each question. As discussed above 
under "format," [contrast the previous short form, with san5)le output in 
Appendix 2>jwA2] we do not use ranked answers or a scale 1 to 5. However, 
many questions do have an implicit rank. Even with "spelled out" answers, 
it would be possible to "score" the responses on a linear scale 1 to 5, 
both for questions where one extreme is "best," and for those where the 
middle is best. Hence it would be possible to compute means and standard 
deviations [a mean without its associated standard deviation is a very 
dangerous itemi]. 

All present computer programs have been written in such a way that 
space is available for storing means and standard deviations. However 
complete programs that compute and print such data have not been written. 
The main programming effort that would be involved is in formatting the 
printout - a tedious but fairly straightforward project. If means and 
standard deviations were computed, we believe that the basic original dis- 
tributions should still be printed out. 



1. 5 Public Display and Distribution 

Questionnaires are distributed towards the end of term. However, no 
output is given to instructors until after all final grades are recorde"!? 
Students are so informed in advance. 

The printed output for all individual sections and for all aggregates is 
put on display in the Physics Building, both for faculty and for teaching 
assistants. [That teaching assistants results be displayed is at the re- 
quest of the Physics Graduate Student Association. Since teaching assistants 
are only apprentice teachers, it would seem perfectly proper not to dis- 
play teaching assistant results.] 
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Each individual teaching assistant or faculty member receives an ini- 
tial copy of his section output with an invitation to point out to the 
program administrators any strange data, or any other possible errors. 
After such errors" are investigated and corrected (generally on matters of 
section number, number of students enrolled, informal team-teaching), the 
tmal computer program is run, and the output distributed. Each person re- 
ceives output on all sections that he is involved in, either as teacher or 
as supervisor of teaching assistants, and all aggregates of which these 
sections form a part. Similar packets go to various course and other depart- 
mental commit^tees.„_X^uch packets always contain a copy of the questfonnaire. ) 

1.6 Comments 



The questionnaires are designed to ask all possible questions. However 
students are also invited to write specific comments on the back of the ans- 
wer sheets. These comments are not "processed" in any way, but are made 
available to the person being assessed, and to the Department chairman or 
his delegate. They are not available to other teaching assistants or faculty. 
. '■^^^^^'^ ^^^^ such a procedure means that the comments do not get in- 

to the public record, and hence into any subsequent evaluative process. Since 
comments are liKely to represent perhaps biassed extremes of opinion, it 
IS perhaps not a bad thing that they in fact become suppressed. 

A variant of the above procedure was tried out for Summer School 1971. 
It consisted of a separate comment sheet with indicated headings (see Appen- 
dix :ip.A21). The intent was to make the comments more accessible by topic 
The actual effect was to encourage "redundant" comments - students tended' 
to repeat m words what they had already marked on the answer sheet. 

It is clear that we have not found a fully satisfactory way to solicit 
and analyse useful comments. 

1.7 Possible Modifications 

At this point no attempt has been made to analyse correlations between 
individual students responses to different questions. However the original 
data is available, so that such programs could be written. [The previous 
short form questionnaire was processed to analyse correlations to a certain 
extent (see sample output. Appendix ^ p. A2) . Conceivably its programming 
could be user-=as a basis for such correlation analysis.] 

Another modification which can be fairly easily programmed has already 
been discussed; namely the use of ranked possible answers on a scale 1 to 5 
and the computation and printing of means and standard deviations. 

A third modification would be to print each question on the print out 
or at least a paraphrase. Unfortunately, the output would then become very 
large. 

A related modification, which can be programmed using the present out- 
put tapes, IS to print out the results question by question rather than sec- 
tion by section and aggregate by aggregate. Thus, for example, one would 
have a page labelled "question 10," followed by the question, and the pos- 
sible answers, followed by the responses of each section and each aggregate. 
Such a style of output <:ouXd be used as well as the present style. It en- 
ables comparisons between individual sections to be made more readily for a 
few key questions. However it is not as useful for obtaining an overall 
view of one section. 



1.10 



In fact, one could suppress the print-out of many of the questions (such 
as the diagnostic or identification questions) or do many other variations 
on the print. One virtue of the present programming of AGGFORM is that 
it is basically a set of subp rograms embedded within the general structure 
of an all-purpose summarizing program called SUMX, used extensively by the 
High Energy Physics research group for making lists, histograms, etc. All 
quantities computed by AGGFORM are stored on the output tape in such a way 
as to make them available for further processing by other subprograms 
(either existing ones or specially written ones) within the SUMX structure. 
Thus many additions to the present system are possible without too great an 
investment in prograimning time. 

The above describe some possible general modifications. Through exper- 
ience, we have also developed a set of prossible minor improvements which per- 
tain to details of the present PATS. These are discussed in Part II in an 
Appendix. 

1.8 Budget 

In Appendix 6,ptA2 we present the budget of PATS over the last year. 
The actual computing time costs do not vary directly with the number of stu- ~ ' 
dents processed, and as yet we do not have sufficient experience to deter- 
mine a good estimate. It should be clear that PATS requires both consider- 
able manpower and funds. In particular, the time and labor involved in pre- 
paring the control cards to. instruct the computer how to form the heirarchies 
of aggregates is considerable. Detailed modifications of the computer pro- 
grams could probably be made to simplify this task somewhat - in particular 
one could hope to prepare a "standard" set of control cards which could be 
used semester after semester with only minor alterations on each use. 

However, while PATS is certainly a sizable and costly endeavor, its 
cost is very small compared with the whole teaching budget of the Department, 
and we believe that its impact on this teaching mission is both salutary and' 
very cost effective. We strongly believe that PATS should be continued in es- 
sentially its present form. 

1.9 (^neral Review and Discussion 

We believe that some of the more important features of PATS are: 

a) different questionnaires for different aspects of the teaching mis- 
sion 

b) rather complete questionnaires 

c) spelled out possible answers (as opposed to ratings on, e.g., a 5 pt. scale) 

d) the display of distributions of responses to each question, together 
with the ability to add means and standard deviations. (But the 
distributions should then still be printed out.) 

e) the computation of heirarchies of aggregates 



1.11 



Not all such features are appropriate for all possible uses. (Even 
for the present use, some modifications are likely in the future ) We will 
be content if the reader simply considers such features and decides for him- 
self if they are appropriate for his purpose. 

The alert reader will notice one glaring omission in the above. Very 
little effort has gone into the problems of validating the questions; testing 
for consistency of responses by individual students (e.g. by repeating es- 
sentially the same question reworded elsewhere); testing for reliability 
(e.g. splitting the responding students at random into two groups and com- 
paring the distributions for the two groups); and in general attempting to 
find out what the results actually mean. Many of the questions have been 
taken with modifications from other questionnaires that were developed and 
very intensively tested and validated, but for other uses at other places, 
e g the campus wide questionnaires developed and used at Princeton Univer- 
sity, and those developed by Milton Hildebrand and Robert C Wilson-, at 
the Davis Campus of the University of California. 

The lack of any serious discussion of validation should not be taken 
to imply that we believe we know how to interpret the results we obtain, 
m fact one of the authors (CK.) has strenuously objected to the use of 
"information" obtained by student questionnaires unless other independent 
sources of information are also systematical ly, used. Tlie Physics and Astron- 
omy Department has instituted a faculty visitation scheme as another facet 
of its overall aim of improving the quality of the courses and teaching. 
In this scheme an individual faculty member in a lecture course in a semes- 
ter is visited by at least two visitors, each of whom works completely in- 
dependently of the other, and makes at least two visits plus having signif- 
icant discussion with the visitee concerning aims, methodology, course 
outline, etc. The primary purpose of this scheme is not to evaluate the 
teacher, but rather to assist him in improving. However, it is clear that 
this scheme will greatly assist the Department in learning how to interpret 
the results of PATS. 

1.10 A Guide to the Overall Operation of PATS 

In this section we summarize and expand upon v;hat has been said prev- 
iously about the overall operation of PATS, adding enough new details as 
necessary to give a fairlv accurate brief portrait of the work involved. 
An even briefer overview can be obtained from Table 2 , which sketches 
the entire process in outline form. The reader should be warned that, par- 
ticularly with regard to the table, some statements may imply less work 
than is actually involved. For example, preparing the control cards for 
AGGFORM is a non-trivial task. Table 2 should not be interpreted as 
giving the best chronological order in which to run, PATS, only as being a 
logical flow chart. 

Packets of questionnaires, standard answer sheets, #2 pencils with 
erasers , and instructions to the instructor (Part II gives details) are dis- 
tributed Jo each instructor, in a large manila envelope upon which is written 
the course number, section number, name of the specific person being evalu- 
ated; and whether lab, recitation, or lecture. There is also a nine digit 
code number written on the envelope, the code corresponding to this informatic 
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TABLE 2, Step-by-Step Operation of PATS 

A. Administering the Questionnaires 

1. Packets of questionnaires and standard answer sheets given to each 
instructor. 

2. Each instructor infornied of the 9-digit code number which identifies 
his section. 

3. Students fill out answer sheets in class, entering code number on 
these sheets. 

4. Filled-out answer sheets turned in to departmental office. 

B. Preparing and Running DATAREAD and INITPRT 

1. Answer sheets converted to IBM cards by Digitek processor. 

2. Header cards prepared for each section (giving instructor's names, 
etc.) and placed immediately before Digitek cards for that section. 

3. Control cards prepared for DATAREAD. 

4. Data processed by DATAREAD. 

5. Corrections made to data as needed, DATAREAD rerun if necessary. 

6. INITPRT takes output of DATAREAD and prepares printout for each sectior 

7. Printout checked for errors, corrections made to data if necessary. 

C. Preparing and Running AGGFORM 

1. It is decided what aggregates will be formed. 

2. Descriptive title and a unique l-to-4 digit number assigned to each 
aggregate. 

3. Number of copies of printout needed for each aggregate is determined. 

4. Control cards prepared for AGGFORM. 

5. AGGFORM reads tape produced by DATAREAD, forms aggregates and prints 
results as directed by control cards. 

6. AGGFORM output checked for errors, particularly in control cards. 
Program rerun as necessary. 

7. The desired number of printed copies of the results for each aggregate 
and for each single section are produced. 

8. Each instructor is given a copy of the results for his section and for 
all aggregates in his appropriate heirarchy. 

9. All single section and all aggregate results are put on public display. 

NOTE. This table shows a "logical" flow chart of operations. In fact many 

of the steps can and should be done in a different temporal order from 
that shown here. 
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The instructor transcribes this information onto the blackboard, and 
distributes all the material. Students enter the information on the black- 
board onto the answer sheets, including the code number, and then answer the 
questions by marking their choice of response on the answer sheet. Filling 
in the answer sheet seems to take typically 15 minutes. All materials ( incl u- 
ding the pencils ) are then collected, and given to a student} to carry to the 
departmental office. (This is to ensure that the person being evaluated 
does HOC see the results or the data before he issues grades). 

The answer sheets are read at the Computer Science Center by the Digitek 
processor, and converted into regular IBM punched cards (typically two cards 
per student answer sheet). The Digitek cards and the answer sheets are re- 
turned to the Department. The answer sheets are saved since they possibly 
have written comments, but all further processing makes use only of the Dig- 
itek cards. Note that in the Digitek process only the responses to the 
questions and the code number get transferred to the Digitek cards, and not 
the other written identifying information. 

The Digitek cards are then assembled with "header" cards in front of the 
cards from each section, and overall "control" cards, to form a complete 
data deck. Header cards describe each section, and contain the code number, 
course number, section number, instructor's name, a possible comment (e.g. 
"Honors," "Team- taught," "seminar, so no questionnaire issued," etc.), and 
the enrolment in the section (allowing for students dropped and withdrawn). 
ITius the header cards serve the purpose of "reinterpreting" the code number 
into recognizable identifying information for this particular section. Con- 
trol cards describe for example the (variable) format of the questionnaires, 
(e.g. "how many questions"), as well as the date (e.g. Spring 1971) and 
other general information (e.g. Physics, Faculty). We divide Physics from 
Astronomy, and Faculty from Teaching Assistants. Thus we have four separate 
data decks and subsequent processings. 

An individual data deck is then processed by the first of three devel- 
oped computer programs, called DATAREAD. This program, section by section, 
reads and prints each Digitek card, tests for and warns of missing or incon- 
sistent code numbers, and sums the responses to each question from that sec- 
tion. DATAREAD also prepares*' this summed data onto a magnetic tape. The 
printed output from DATAREAD is checked, in particular with reference to the 
code numbers. At this time one can go back to the original answer sheets 
(with the written identifying information) in order to check these apparent 
errors. Some students fill in their own social security ID instead of the 
regular code, and some leave it blank - both of these ca:ise no difficulty 
and need not be traced dovm in general. But sometimes one finds indications 
that answer sheets from two sections have gotten into one envelope and hence 
one section's digitek deck; or similarly the answer sheets for two different 
questionnaires from the same section, e.g. where a teaching assistant teaches 
both a recitation and a lab to the same students . Sometimes this is when 
unofficial team teaching comes to light. Digitek cards can then be physically 
moved from one section to another within the overall deck. 
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After such corrections are made, a second program INITPRT (initial print) 
prints out the distributions of responses st-tion by section. This output is 
distributed to the individuals to whom they refer, who are invited to scru- 
tinize it for further errors [typically in enrolment numbers]. Any sucli 
errors are corrected in the original data decks, and a final riin of DATAREAD 
IS made. Tais run produces a magnetic tape which then becomes the input to 
the final program AGGFORM (aggregate formation). This program computes all 
aggregate distributions and prints out copies according to a variable print 
scheme (so that enough copies of higher aggregates are printed as needed), 
nie aggregate distributions remain available in the form of tape output for 
turther processing. It can also print out copies of the individual section 
data. 

Since AGGFORM can form many different aggregates at many different levels, 
it requires a more complex set of control cards than the previous programs 
it 'fJ^li precisely what to do. The first step In preparing to run AGGFORM 
is, fairly obviously, to decide what aggregates one wishes to form. Some 
guidelines for this decision-making are given in the chapter on AGGFORM in 
Part II. 



..ach aggregate is given a title (e.g., "Physics 10 and 11 Lectures") 
which will be printed at the top of the results for that aggregate, and is 
assigned a number from 1 to 9999, each aggregate having its own distinct 
aggregate number." Later, when the results are printed out by AGGFORM 
they will appear in order of their aggregate number. It is not necessary to 
create the aggregates in the same order in which one wants them printed out; 
m fact, it is sometimes even Impossible or unwise to do so, owing to the 
structure of the program. This results from the fact that AGGFORM is actually 
run In stages, each stage building on the results of the previous stage. Thus 
aggregates of different levels are often formed in separate stages, yet one 
might like to see them next to each other on say, the final list of results 
to be put on-public display. 

One next figures out how many copies of the aggregates he will need for 
distribution to the individual instructors. It is clear that if each instruc- 
tor is to be given a copy of every aggregate of which his section is a part, 
then we need more copies of those aggregates involving many sections (and 
thus many instructors) than of those involving just a few. (Altering the pro- 
gram so that it automatically computes this figure is one obvious minor change 
tnat could be made to the present version of PATS.) 

Control cards are then made up for each aggregate, specifying the title 
number of copies desired, additional comments (if any) to be put on the prin-* 
ted output, etc. There are also control cards to tell the program which 
sections (or aggregates from previous stages) to include in forming each new 
aggregate. A considerable amount of time and care must go into the prepara- 
tion oj. all of these control cards. 

Normally one then runs AGGFORM with only a request for one copy of each 
aggregate, not for the multiple copies for distribution. Almost inevitably 
there are mistakes in the data or control cards which must then be corrected. 
When all such mistakes have been corrected, the program is run a final time, 
this time generating all the printout needed. The program operates in such a 
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way that once it has been completely run and a listing of all aggregates (or 
single sections) has been generated, additional printed copies of this out- 
put (both aggregates and single sections) can be obtained very easily. All 
of this Is explained in more detail in chapters 2 and 5, along with additional 
controls which the user has over the nature of the printout. 



CHAPTER 2. FURTHER DISCUSSION OF THE "HEIRARCHY OF AGGREGATES^' CONCEPT 

2. 1 Introduction 

In this chapter, we dlscvHS those features of PATS which would apply to 
any questionnaire procedure tuat incorporates the general concept of heirar- 
chies of aggregates. There are many problems that must b& solved in any such 
scheme. We hope that a presentation of "solutions" used in PATS will pro- 
vide the reader sufficient information so that he can develop his own soltt* 
tii^s to his problems. Thus this chapter is still aimed at the generality of 
questionnaire procedures , and belongs to Part I of this repoit. The detailed 
maniial for PATS which implements these solutions appears as Chapter 5 ot 
Part II of this report. That chapter commences with ^ summary of this chap- 
ter so that it is Itself self**contained. 

2.2 Hov Do Yog Make an Aggregate? 

As stated in Chapter I, the purpcae in making aggregates by "combining" 
individual sections Is to make comparisons easier and more meaningful* and 
to make the reams of data easier to interpret. Naturally the question of 
which aggregates to form* and what to include in each one, is left up to the 
user. The program is completely flexible in this regard. The di«;cussion 
in Cliapter 1 should have provided a .general idea of how one might choose the 
contents of such aggregates and heir/irchles of aggregates. PATS does follow 
a set of guidelines which are discuFsed in Chapter 5. However » we must still 
deal with the basic question of ho^^ actually to combine the results of differ-* 
ent sections in one aggregate. 

Th€ need is to find a way of forming aggregates so that they summar** 
ize in a reasonable fashion the information in their component parts. Ob- 
vious ly» each question must be treated individually * usually one cannot 
sensibly combine the results of two different questions. 

As noted earH4r« the output for each aggregate is in the form of a sim*- 
pie listing of the distribution of responses to each question, that is, how 
many students (or what percentage) chose each of the possible answers. To 
obtain these distributions for a single section, one merely adds the results 
from all the individual student answer sheets. Thus one knows how many stu- 
dents chose, for exanq>le, answer "d" to question #21, and upon dividing by 
the number of students who completed the answer sheets, one has the percentage 
distributions. This is precisely what DATAREAD and INITPRT do, for the sin- 
gle sections. 

When, however, one tries to combine two or more different sections, it 
becomes clear that merely adding all individual answer sheets together (and 
dividing by the total number of students involved to get percentages) is cer- 
tainly not the only way, and probably not the best way to proceed. An aggre- 
gate including AGO students could be made up of one section t^ith 250 students, 
one with 125 students, and one with 25 students. If we proceed as sketched 
above we do, in fact, weight individual students* opinions equally, but we 
don't learn very much about the relative teaching ability of the three in- 
structors involved, since the results are dominated by one section. To com- 
pare instructors, then, we must be able to v/eight instructors equally. For 



faculty, most (if not all) of whom teach only one section, this means 
weighting sections (e.g., lectures) equally. But T/As usually teach more 
than one section. For T/As then, we must first fona an aggregate of all 
sections taught by a given T/A, (called a "complete description" of that T/A) , 
and then be able to weight that aggregate equally with other such aggregates. 
Since some T/As teach only one section, we must.be able to combine such sin- 
gle sections with the aggregates (complete descriptions) for those T/As 
who teach more than one section, and still wei^t all T/As equally. To sim- 
plify the phraseology, we will use the term "component" to designate one of 
the various entities - single sections or aggregates - which are put together 
("aggregated") to form a given aggregate. 

Thus we can phrase our conclusion as follows: we need three types of 
weighting in forming aggregates - by student, by section, and by component. 
Which weighting is most appropriate is not always clear; probably one. form 
of weighting is best for some questions, another form for others. It may 
also depend somewhat upon the specific use to be made of the results. In 
any case, it was decided to do weighting by student and by section for all 
aggregates, and to do weighting by component whenever necessary to obtain / 
equal weighting by instructor. Resulting distributions are given in terms' 
of percentages. In fact, the prograir/s ability to weight different components 
equally extends beyond the specific case mentioned above, where each compo- 
nent in question represents the responses for one and only one instrucrtor* 
Actually, the program can weight any two (or more) components equally. For 
example, consider the aggregate: "all physics mainstream courses." iJe could 
if we wished obtain this aggregate by weighting equally the two fpilowing 
components - the aggregate of all undergraduate physics major courses, and 
the aggregate of all graduate physics courses. As yet it is n^t clear what, 
purpose, if any, would be served by this. ^ 

2.3 The Problem of Team-Teaching; Combinations 

Sometimes two instructors share teaching responsibilities for the same 
section. We handle this situation by asking the students to complete two 
questionnaires, one for each instructor. This is fine as long as we just 
consider,, the single section summaries. When, however, we prepare to make 
aggregates of this section together with, say, two others, we are faced with 
a problem of unequal weighting: the team-taught section appears t wice . We 
can overcome this dilemma by combining the two sets of questionnaire results 
(for the two separate instructors of this one section) into a pseudo-single 
section (taught by a pseudo-single instructor). This is a special type of 
aggregate and will be referred to as a "combination." The simplest v>ay to 
handle this is simply to average the two sets of results. If, as can and 
does happen, we gt t different numbers of completed answer sheets for the two 
instructors, we set the number of responses for the combination section equal 
to the average of the corresponding numbers for the two separate sets. 

2.4 The Probler if Low Response Levels 

Suppose, in a section of 25 students, 13 students tutn in answer sheets 
but two of these 13 sheets are blank (i.e., the students respond with absten- 
tions). Are the section results meaningful enough to be printed out? Should 



this section be included in any aggregates? These are the questions we 
consider in the following paragraphs* 

Take first the case of single sections. One could, for instance, 
attach a "confidence level" to the results which reflects the percentage 
response* A section with low response would have a low confidence level. 
Or one could divide up the response rercentages into, say, three categories - 
good response, borderline response, insufficient response; within a given 
category all response levels would be treated equally. This latter alter- 
native was the one chosen. For the first category, the results are printed 
with no further comment; for the second group, a warning is given. The 
third group is considered to be rejected, and even though results are still 
printed, they are preceeded by a second, stronger warning and are not inclu- 
ded in any aggregates imless they are forced in by the user. The actual 
cutoff percentage values for both warning and rejection are set by the user; 
the quantity tested is the ratio of the number of completed answer sheets 
to the number of students enrolled in the section. At present the cutoff 
levels are set at less than 50% response for rejection, between 50 and 70% 
for warning. 

The reason we consider an individual section as "rejected" if the par- 
ticipation rate is less than 50% is, of course, that it is possible that 
the respondents do not- jEorm a truly representative sample of all the students 
In that section. We must admit the possibility that this data from this 
section is biassed. Since there is no way to test if it is indeed biassed, 
we are forced to consider that each responding student is a member of a 
prima facie bi assed sample ef the original section population. 

We nex.c consider the case of aggregates. At first sight, when forming 
the distribution w^ich weights students equally, one might decide to include 
in the aggregate all completed answer sheets, even those from sections which 
were themselves rejected. Then we could divide the number included by the 
total number of students enrolled to see whether the percentage response was 
sufficiently high to justify making the aggregation. However, for the dis- 
tribution which weights sections equally, we would presumably only wish to 
include uon-rejected sections , and then the cutoff would be based on the 
munber of included sections divided by the total number of sections which 
meet the criteria for being Included in the aggregate in question. And, 
correspondingly, there v7ould be a third type of cutoff for weighting by comr 
ponent. 

With three such different cutoffs, different parts of the data would 
be included in the different weightings. Each responding student would be 
included in the weighted-by-student aggregate distribution; but only those 
students who were part of non-rejected sections would be included in the 
•Veighted-by-section" aggregate distribution, etc. Thus, it would be impos- 
sible to say uniquely just what data an aggregate included. 

We argue that such a scheme is not only complicated and ill-defined; but, 
far more important, that it is inherently subject to bias in the sense of 
possible sampling errnr. Particul^iriy \itiea considering an aggregate such as 



one made up of all recitation sections of a large course, it is important 
to realize that the distribution of individual students into individual 
sections which meet at different hours is likely to be highly non-random > 
since it is determined by student schedules, an<f different groups of students 
will be taking different other courses around which they must build their 
schedules. 

For this reason, we decided always |to work in terms of components . For 
faculty lecture sections, each section i's always considered an equal com- 
ponent of the final aggregate. A component which is a section is "tested" 
for prima facie bias by using the cutoff values as discussed above. If it 
is "rejected," no data from that component is included in computing the ag- 
gregate. An aggregate is itself tested for prima facie bias by use of the 
ratio of the number of components actually accepted to the number of compo- 
nents ab initio eligible for inclusion in the aggregate. 

For recitation and laboratory sections taught by T/As, an aggregate 
"all recitation sections of course X" is considered to be made up of com- 
ponents which each represent a complete description of an individual T/A. 
These ab initio eligible components are tested for inclusion in a manner sim- 
ilar to that discussed above. Thus all data for a particular T/A is exclu- 
ded if the number of his accepted individual sections is no greater than 50% 
of the number of his ab initio eligible sections. 

We should emphasize that this "solution" to the question of possible 
sampling error is not the only way in which one could proceed. But some 
solution to the problem must be considered in any such scheme (if only by 
omission). 

Distributions of responses are normally not printed out for rejected 
aggregates, unlike the case of rejected single sections. The rationale 
in the latter case is that single section distributions are still close enough 
to the raw data for there to be some utility in their being printed, even 
when student response has been insufficient to warrant their being included 
in any aggregates. Rejected aggregates are simply indicated by their titles, 
together with a list of the accepted and rejected components and a statement 
that the aggregate was rejected. 

The distributions that are printed out for aggregates are in terms of 
percentages only. As soon as one begins weighting the single section results 
in various ways, it ceases to make sense to print out the "nusober of students 
who chose answer C to question 21." This nimiber only really exists when 
students are weighted equally, not when sections or components are weighted 
equally. Since one cannot print such numbers for all weigjhtings, we choose 
not to do it for any. 

It is very Inqportant to remeinber that these percentages refer to the 
number of students responding (without abstaining) in the included components. 
Thus in the aggregate of all Physics 27 recitation sections, the response to 
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answer question 21 may be shown as 25%. This means that 25% of the students 
responding in the included components chose this answer. It does not mean 
that 25% of the students enrolled in these sections felt a certain way, and 
more importantly, it does not mean that 25% of all students enrolled in Physics 
27 responded (or would have responded) in this way. In fact, the response 
in the rejected sections may have been quite different. One should be care- 
ful to keep in mind at all times exactly what each aggregate does and does 
not represent. It is quite probable that there is a strong correlation be- 
tween whether a person completes the answer sheet or not, and the answers he 
would give if he did . That is, the type of person who would fail to complete 
the answer sheet might, be the very person who would have answered the questions 
in quite a different manner from those students who did complete answer sheets. 
This brings an extra uncertainty to the question: to what extent does an 
aggregate really represent the feelings of all students enrolled in a course 
or group of courses? It is for this reason that warnings are printed, and 
sections and aggregates with insufficient response are rejected. 

2.5 The Effect of Cutoffs on the Interpretation of Aggregates 

It is iiiq>ortant to be aware of the effect that the cutoffs described 
in Section 2.4 have on the contents of the aggregates, and thus on the 
meaning which can be attached to them. Sinq)ly stated, it means that an 
aggregate reflects only the opinions of students who were in non-rejected 
components. The following example should illustrate this point; it has been 
made lengthy so that the complexity of the situation can become apparent* 

Consider a hypothetical Physics 27 course, with 15 recitations taught 
by 5 T/As, whom we shall call V, W, X, Y, and Z. Our first step in forming 
aggregates is to make a complete description for each T/A. Mr. V teaches 
only one section, so we're through with him for the moment. Mr. W teaches 
four sections; each of these is then "eligible" for inclusion in his com- 
plete description. First , howevei^ we must check the response level for each 
section; any section with fewer than 50% of the enrolled students comple- 
ting the questionnaire (without abstaining) is a "rejected" section and will 
not be included in this complete description. If 50% or more of these sec- 
tions are rejected (i.e., two or more in this case), Mr. W's complete descrip- 
tion will not be computed ; it will be a "rejected aggregate." Note that as 
AGGFOBM is now set up, an even 50% of students enrolled in a section is suf- 
ficient to keep that section from being rejected, but this is not true of 
aggregates. In the example, Mr. W's complete description will not be compu- 
ted if two and only two of his four sections are accepted. (The user, of 
course, has the option of changing the cutoff. He also has an "override" 
feature available which allows him to force the acceptance of particular com- 
ponents in instances such as this one.) We continue this way with the other 
T/As and might end up with the following results: 

T/A No. of Sections No. of Rejected Sections Aggregate Formed 

W 4 2 No 



X 



4 



0 



Yes 



Y 



3 



2 



No 



Z 



3 



1 



Yes 



Thtis only one of these complete descriptions is "complete** in the sense 
of including all of the eligible sections. Two are rejected (W and Y) and 
the last (Z) gets printed out with a warning, since fewer than 70% of the 
eligible sections are included* As for response levels, consider Mr. Z's 
complete description. In the worst possible case, each of these sections 
might have exactly 50% response* If all three of Mr. Z's sections halve the 
same enrollment, then Mr* Z*s "complete description** will include responses 
from only one- third of his students I Tliis Illustrates the necessity of 
the warning* 

Now let us form the aggregate of all recitation sections for this course. 
For such a case the basic components are taken to be the T/A complete descrip- 
tions* We now have five ab initio eligible components, one for each T/A: 
the four complete descriptions listed above, and the single section taught by 
Mr* V (which does represent his complete description). We will assume that 
Mr. V's section has greater than 50% response. Thus out of five ab initio 
eligible components V, W, X, Y, and Z, we have three acceptable components 
V, X, and Z. The relevant ratio is 3/5 = 60% which passes the acceptance 
criterion, but with a warning. Hence, this aggregate will be formed, and a 
warning will be printed with the results. Note that at this stage we only 
have seven of the original fifteen sections left, and presumably an even smal- 
ler percent of the students enrolled in the course. One can imagine situa- 
tions which would be even worse. The need for procedures which encourage 
high levels of response should be obvious. 

By now it should be clear that our particiilar "solution" to the problem 
of sampling bias has serious implications. Other "solutions" are possible. 
In this particular example, it is perhaps tempting to decide that the overall 
aggregate should be considered to be made up of components which represent 
the individual seccions. In that case, all non-rejected individual sections 
would be included in the overall aggregate, so that data from ten out of fif- 
teen sections would be included (instead of the previous seven out of fifteen). 
There would be a distribution "each included student weighted equally", and one 
for "each included section weighted equally." There would not be a distri- 
bution "each included T/A complete description weighted equally." 

We chose not to proceed in such a manner. But the user of the program 
A6GF0RM does have such a possibility available to him, since a little thought 
shows that what is critical is the specification of the components that make 
up a particular aggregate. The component specification "complete descriptions 
of T/As V, W, X, Y and Z" produces our scheme; the component specification 
"all individual sections taught by T/As V, W, X, Y and Z" produces the alter- 
native scheme. The individual user must simply decide which scheme seems 
more valid to him. [The present program AGGFORM does not have the capability 
of computing a distribution "each included T/A weighted equally" which includes 
all non- rejected individual sections. 

2.6 Conclusion 

It is hoped that this chapter will have raised the types of questions that 
must be considered in any scheme that makes use of the concept of heirarchies 
of aggregates. Some of these questions also arise in schemes which simply 
produce only a single "overall average distribution." The problem of possible 
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APPENDIX 1, THE FALL 1970 SHORT FORM QUESTIONNAIRE 
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WARTKEyr OF WIYSIC5 ANT) ASTRONOWf 
imiVKKSITV 0*- y>KYLA.SO 
Coltrtc I'aik. M:iryl.ina 



Tht5 qncstlonnjirc whtch you arc a^kcJ to comj>letc Is of the hlehe'it 
i^riancc. Fron it the chatrcwn oC tlic [Knartornt of I'hysics antl Astronony 
will obtain the inforrtiticn about tcathins q<uility fiow whuh rtcomt-nUat I'.n- 
vfil be fmUe foe the schrUuIin^: of courses. TJie Intorouticn will .tI-.c» jo an 
la^rt.int consideration In the ev.ilu.ition am: pronotirn of faculty ror/oLrs 
•«l Will be u-.Cii to evaUatc the tcr.ci.1n5 -^-n^r o( graduate cssi .:T\ts. 

finally c.ich tcjcher will obtain valuable liifornation about how he appears to 
>«. 

Before you begin read the ettt ire gncntirrnaife carefully and answer the 
following quc5tion: 

Do you think this questionnaire asks questions about what ^^ou think Yes Vo 
•re the K»st icportant points about the teacher? j^*^ j 1 

. Part I 

In all of the questions coppare your teacher in this course with all of the 
science and mth teachers you ever haa or ha-*, new. Inacinc that icr each 
question you vrould give your test teachcrj't^e ^rade of 4 and the worst teachers 
•a 0. On this scale ?ivc your teacher in tl:i«; course a 4, 3, 2. I. or 0. If 
joa don't feel that you can aake a rea>cnaL-lc ccrparison. check tne "can*t ccrpjr." 
box. (Note: The word *'irc" below ctcans "He cr She") 



1. He had excellent cnn^nd rf the subject a»t£er as far as I can tell. 

2. He vas usually veil prepared for class. 

3. Hfc held the aLtcntion and interef;;. of the clas^* 

4. He tried to ttake nc understattd rather than neri^^rise. 



He succeeded in caking ne understand vliea 1 put forth vhat 1 :hlnk 
vas a reasonable effort also. 



6. He gave honevork which tarir.ht ric a great deal. 

7. He gave about tl«c rij^C ar:i>urt of Uorcwofk. 
8* He gave exxis vfiich were "fair.** 

9. He gave exan«; from which X learned a great deal. 

10. He gave pronpt feedback on gradud uierial* 

11* He vas fair in die grading tnat he did or supervised. 

12. He encouraged b« to see hln outslie class when I needed help. 

13. He encouraged questions in cla^s. 

14. Re was patient %^tn X asked • question. 
IS. 



16. 



He was an "inspiring*' teacher. (Lse your own interpretation of 
this word.) 

He seencd to "knew" nost of tht po»>!>le in the class.^ (Answer 
only If your clajs had fewer thjn ?0 people in it»> 



Tart TI 

Hbtf think back o\er all of Fart X and answer the following: 

17. If I were to take another physics course .In which this person was 
teaching* X would: 



(a) seek his section out. 

(b) avoid his section. 

(c) X have no strong feelings. 

18. Should this person be retained iti this teaching capacity? 

(a) Ko, 

(b) Yet. 

(c) t bave ^pinien* 



/ 



432 10 



Can't 
Cor4*acc 



L3:r--r.::.i 



•V" 



nVij.i":.. '» 



I_ElI__j : 

i I I l-L.!— t 
till'' i 

UJlIII-J 



(•) lh 

(b) □ 

(O □ 



(•) a 
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APPENDIX 3. PARTS OF THE SEVEN QUESTIONNAIRES PRESENTLY USED BY PATS 



3a. The Complete Form 1 - "General" Courses (Lecturer) 

UNIVERSITY OF MARYLAND 

Physics Department 
Teaching Questionnaire 

Form 1 - "General" Courses (Lecturer) 



The purpose of this questionnaire is to obtain in an anonymous fashion your 
opinion about the physics and astronomy teachers and courses. This infor- 
mation will be used in many ways — to improve each teacher^s performance, 
to help the chairman assign faculty members to courses, and to provide 
important input in the evaluation of faculty with respect to retention and 
promotion. 

Do not put your student number in the box labelled student number in the lower 
right hand corner of your answer sheet. Instead put the code number for this 
course (which should be on the blackboard) in that nine digit space by block- 
ing In the appropriate blocks, e.g. Code Number 012210021 would become: 
Your instructor will get the tabulated results of this V^TuorHT 
questionnaire, together with any written comments, 
after the final grades in this course have been 
submitted. 



NUMBER 



T 



J 



General Instructions 

1. All entries must be on the STANDARD ANSWER SHEET. J 
Comments must be written on the back of this sheet. 

2. Use a Number 2 pencil only . Each question must have 
either one block only or no block marked. Multiple 
answers are not allowed, so that a wrong answer ! 
must be erased carefully and completely. To mark 
the response (f) leave all spaces blank. This will 
be counted. 
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3. Black in your course code number in the nine digit space la3elled"Student 
Number" on your Standard Answer Sheet. 

4. Finally, please write on the top of the Standard Answer Sheet the course 
number; section number; name of the specific person whom you are evaluating; 
and whether this is a lab, recitation, or lecture; (e.g., ••Phys 100, 
Section 201, John Doe, Lab.") This information should also be on the 
blackboard. 



Even if you do not fill out the questionnaire please carry out instructions 3 

4 , and return the otherwise blank answer sheet. Thank you for your 
cooperation. 



II 



Questions; 



!• With regard to this questionnaire: 

basicaUy approve of such questionnaires, and hence you arc re- 
spending to the vhole questionnaire. ^ 
You fuid such questionnaires an laposltlon, but In spite of this view 
you are responding to the vholc questionnaire. ' 

kllsJ^*!^*!:''!!^^/^^^^'^*^^^^^ '""^ questionnaires, and hence 
(PUASE, after filling i„ the code number) intend to obstiln completely. 



(f) 



The lecturer presenter the historical origins of ideas and concepts • 



(a) most of the time 

(b) some of the time 

(c) very infrequently 

(d) never 

(f) no answer (i.e. leave blank) 



3. In reference to question 2, I prefer - 

(a) Much more 

(b) more 

(c) about right 

(d) less 

(e) much less 

(f) no answer 



4. The lecturer related physics to other disciplines - 

(a) very infrequently 

(b) some of the time 

(c) never 

(d) most of the time 

(f) no answer (i.e. leave blank) 

5. In rfftrtnca to question 4, I prefer • 

(a) much more 

(b) more 

(c) about right 

(d) less 

(e) much less 

(f) no answer 

6. The lecturer brought out the "philosophicar* beauty in the subject, i.e 
soM feeling for the internal symmetry and logic of the subject - 

(a) Bost of the time 

(b) some of the time 

(c) very infrequently ^ 

(d) never 

(f ) no answer (i.e. leave blank) 
/. in reference to question 6, I prefer - 

(a) much more 

(b) more 

(c) about right 

(d) less 

(e) much less 

(f) no answer 

a. The lecturer related class topics to students' lives and concern and 
more generally to the outside world - concern, and 

(a) a lot of the time 

(b) very infrequently 

(c) some of the time 

(d) never 

(e) as often as possible 

(f) no answer 
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9. In r«fertnce to question I. I prefer 



10. 



(•) 

(b) 
(c) 
W) 
(t) 
(f) 



Much More 
•ore 

about right 
less 

mich less 
no answer 



Due to my taking this course* my interest and appreciation for the subject - 

(•) 

(b) 
(c) 
W) 
(e) 
(f) 



have increased greatly 

have increased a bit 

have remained unchanged 

have decreased somewhat 

have been completely turned off 

no answer 



11. The lecturer and course stimulated my intellectual curiosity and provoked 
independent thinking • ' r 

(a) a great deal 

(b) quite a lot 

(c) a little 

(d) not at all 

(e) had negative effect 

(f) no answer 

12. I evaluate the lecturer«s knowledge of the subject as being - 



(a) excellent (b) good (e) 

(d) less than satisfactory (^) 
(f) no answer 



satisfactory 
incompetent 



IS. His ability to get the material across to me is • 



(a) excellent 

(d) poor 



(b) good 
(e) x#ro 



(c) satisfactory 
(f) no answer 



Ik. Hie prtparmtlom for elmaa la 



excellent 
pw 



good (e) 
BO answer 



•atlaflactory (d) leaa than aatlaf^ctory 



ERIC 



15. the lecturer - 

(a) usually started and ended his class on time 

(b) sometimes started late or ended late 

(e) frequently started late or ended late 

(d) frequently started late and also ran over 

(c) often called off class or didn't show up 

(f) no answer 

16. Ihe course as a whole was planned and organized - 

(a) extremely well 

(b) well 

(c) acceptably 

(d) rather poorly 

(e) extremely poorly 
if) no answer 

17. In my opinion the inclusion in the course of lecture demonstrations, 
films » and visual aids was - 



(•) 

(b) 
(c) 
W) 
(•) 
(f) 



too much 

good 

adequate 

not enough 
too little 
no answer 
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9* In reference to question I prefer - 

(a) Mich More 

(b) More 

(c) about right 

(d) less 

(e) mich less 

(f ) no answer 

10. Ihie to mf taking this course, ny interest and appreciation for the subject - 

(a) have increased greatly 

(b) have increased a bit 

(c) have renained unchanged 

(d) have decreased somewhat 

(e) have been coR^letely turned off 

(f) no answer 



11. The lecturer and course stimulated ay intellectual curiosity and orovokcd 
independent thinking - ^ 

(a) a great deal 

(b) quite a lot 

(c) a little 

(d) not at all 

(e) had negative effect 

(f) no answer 

12. I evaluate the lecturer^s knowledge of the subject as being - 

(a) excellent (b) good (c) satisfactory 

W less than satisfactory (4) inconetent 

(f) no answer 

IS. His ability to get the material across to me is - 

(a) excellent (b) good > (c) satisfactory 

(d) poor (e) ziro (f) no answer 



Ik. lia preparation for elaca la 

m) exeeXletit (b) good (c) aatlaftictory (d) leaa than tatltfaetory 
e) poor (f) no anawtr 

IS* The lecturer - 

(a) usually started and ended his class on time 

(b) sometimes started late or ended late 

(c) frequently started late or ended late 

(d) frequently started late and also ran over 

(c) often called off class or didn't show up 
(f) no answer 

16* The course as a whole was planned and organized - 

(a) extremely well 

(b) well 

(e) acceptably 

(d) rather poorly 

(e) extremely poorV 

(f) no answer 

17* In my opinion the inclusion in the course of . ^ demonstrations* 
films, and visual aids was • 

(a) too much 

(b) good 

(c) adequate 

(d) not enough 

(e) too little 

(f) no answer 



11. The lecturcr*s apparent concern in teaching this class showed that he was 

(a) exceedinily interested (b) usually quite interested 

(c) fairl/ interested (d) only occasional!/ interested 
(e) bored ^ (f) no answer 

19. His attitude toward questions in class was - 

i^l encouraging (b) usually encouraging (c) adequately recepti 

(d) flippant (e) discouraging (f) i have no opinion 



20* From Wf experience* the tead)erf,s availability for questions outside 
of class hours was • 

(a) encouraging and generous of his tine 

(b) usually available 

(c) barely adequate 

(d) discouraging and usually unavailable 

(e) always unavailable 

(f) I have no opinion 

21. Ihe feeling between tne lecturer and the students was - 

(a) that of strong goodwill (b) that cf goodwill (c) neutral 
(d) negative (e) antagonistic (f) lo answer 

22 • Ihe exaju - 

(a) always covered aaterial or techniques enphasized in the course 

(b) usually covered Mterial or techniques eaphasized in the course 

(c) soMtiMS covered material or techniques emphasised in the course 

(d) frequently covered tutorial not emphasized in the course 

(e) were totally unrelated to the course 

(f) no answer 

the exams and grading in the course wore • 

(a) much too easy; eesy to get good grade without adequate understanding 

(b) a little too easy 

(c) very fair 

(d) somewhat too difficult to get a good gra^ 

(e) much too hard 

(f) no answer 

I* Ihe assignments in the course were - 

(a) much too hard or much too long 

(b) generally somewhat too difficult or somewhat too long 

(c) about right for the course 

(d) not quite adequate 

(e) non-existent or grossly inadequate 

(f) no answer 

The problems and/or questions assigned were - 

(a) very useftil in understanding the material 

(b) helpful in understanding the material 

(c) somewhat helpful in understanding the material 

(d) generaUy not well selected 

(e) poorly selected 
(O no answer 

• The text(s)« K'uidouts* readings^ etc* were • 



(a) excellent (b) toed (c) satisfactory 

(d) less than satisfactory (e) po^r (f) no answer 
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27. 



l^^e^i^which ^^^^^^^ tpproxinately the 

levei xn which this person was teaching, you would . 



same 



(a) 
Cb) 
(c) 
(d) 
(e) 
(f) 



eagerly seek his section out 

take his section if convenient 

have no strong feelings 

avoid his section if easily possible 

avoid him like the plague 

no answer 



28. On the basis of this person's teaching in this course, I think that he 

(a) merits appropriate recognition for his truly outstanding teaching 
Cb) is a good teacher 

(c) has the potential of becoming a good teacher 

(d) is an adequate teacher, but without - h potential 

(e) should not be retained as a teacher \ this course 

(f) no answer 

29. The math and/or ph/sics background that I assumed was needed in this 
course was - 



31. 



(a) 
Cb) 
(c) 
(d) 
(e) 
(f) 



sufficient to handle the course 

inadequate, but the instructor filled in the gaps 

inadequate and further, the instructor failed to fill in the gaps 

greater than needed 

poorly related to the course 

no answer 



30. In my opinion the general level and speed of the course were 

(a) much too hard and/or much too fast 

(b) somewhat too hard and/or somewhat too fast 

(c) just right 

W) a little too easy and/or a little too slow 

(e) much too easy and trivial 

(f) no answer * 



His use of examples and iTlust rat ions was 

(«) excellent and effective 
(c) adequate and helpful 
(e) noticeably lacking 



(b) good 

(d) less than satisfactory 
(f) no answer 



For epch of the following possible bad habits, wark'block (a) if and only i f 
the instructor suffers from it, and it would be desirable for him to correct 
it; otherwise make no mark - 



32. 

33. 

34. 

35. 

36. 

37. 

38. 

39. 

40. 

41 

42. 



inaudible 

writing was illegible 

spoke either too fast or too slow 

monotonous style of speaking 

poor organization on blackboard 

lectured to the blackboard 

paced too m^ich 

••hums" and '.laws" 

distracting or nervous mannerisms 

caustic or sarcastic manner 

any other bad habits? Mark (a) if 

"yes" and specify on the reverse 

side of answer sheet 



(a) should be corrected 

(a) should be corrected 

(a) should be corrected 

(a) should be corrected 

(a) should be corrected 

(a) should be corrected 

(a) should be corrected 

Ca) should be corrected 

C«) should be corrected 

(a) should be corrected 

M yes 



r4 



For each of the following positive qualities, mark block (a) if the instructor 
has it to a high degree - 



43. 
44. 
45. 
46. 
47. 
48. 
49. 
50. 
51. 



52 



is articulate (a) 
has excellent organization on blackboard (a) 



has very well paced lecture (a) 

has a lively style of speaking (a) 

has cheerful or pleasant disposition (a) 

speaks directly to the class (a) 

has a sense of humor when called for (a) 

he motivated me to do my best work (a) 
he discussed his actions, decisions, 

and selection of topics (a) 
he made difficult topics easier to 

understand (g) 



yes 
yes 
yes 
yes 
yes 
yes 
yes 
yes 

yes 

yes 
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53. 
54. 

55. 
56. 



he sumarized major points 
he stated objectives for each class 
section (a) 
he discussed practical applications. (a) 
Any other outstanding positive qualities? 
Mark block (a) if yes, and specify on (a) yes 
the reverse side of a: swer sheet 



(a) yes 



yes 
yes 



58. 



SriHfa)^^ nUJ' ^^^^ queatlonsT If yes, 

-ark (a) If not, «rk (b) and cpeclfy on the reverse side of aJsJI^r sheet. 



yes 



,bj no (specify on the reverse side of answer sheet), 
X) I prefer not to answer. 

anything you partlculBrlv 
llked^aut the course or particularly dl.llKed. "Si^^^ 

(a) r«s (f) no (I.e. leave blank) 



59. Your class is - 

(a) f xeshaan (b ) sophoaore 

(d) senior (e) graduate or special student 

60. Your aajor is in the area of - 



(c) Junior 

(f) I prefer not to 



nswer 



(a) huaanities (b) social sciences (c) life sciences 

(d) physical sciences (e) other (nark here, and specify on the reverse 
side of answer sheet) (f) I prefer not to answer * 

61* How aany hours of work outside class did you put in on this course? 
(exclude hours related specifically to lab) 

(a) between 0 and 2 hours per week 

(b) between 2 and 4 hours per week 

(c) between 4 and 6 hours per week 

(d) between 6 and 8 hours per week 

(e) over S hours per week 

(f) I prefer not to answer 



62. In the course do you expect to get 



(a) 

(c) 



a good grade 
a poor grade 



(b) an average grade 

(d) I prefer not to answer 



63. How aany classes of this instructor have your aissed 



(a) 

(d) 



none 
■any 



(b) 
(e) 



very few 
nearly all 



(c) 
(£) 



few 

I prefer not to answer 



64. You are taking this course - 



(a) to fulfill a General Education requirement 

(b) to fulfill a requirement of my program 

(c) elective* chosen because of my interest in the subject 

(d) elective* chosen because I needed an easy course 

(•} elective* chosen because I heard the course or professor was good 

(f) I prefer not to answer 

65. Of the courses that would have served ay purpose* this one was - 

(a) only choice that fulfilled ay needs 

(b) my first choice 

(c) my second dioice 

(d) only course still open 

(f) I prefer not to answer (i.e. leave blank) 



All 



3b. 



Part of Form 3 - "Physics" (or G>P.S>) Graduate or Under- 
graduate Courses (Lecturer) 
Questions; 

!• With regard to this questionnaire: 

(a) You basically approve of sueh questionnaires, and hence you are 
respondinf^ to the vhble questionnaire* 

(b) You find such questionnaires an laposition, but in spite of this 
view, you are responding to the whole questionnaire. 

(f) (i»e» leave blank] You disapprove of such questionnaires, and 

hence (PLEASE, after filling in the code number) intend to abstain 
completely* 



3.2 



2. I evaluate the lectnrp-p'c i ^ ^ ^ 

lecturer s knowledge of the subject as boi 

(a) excellent 



(d) less rh... sati'sfncoo- Co} S.pcJt^ satisfactory 
11^ no answer 

His ability to pet the n:atcrial across to me is - 



(a) excellent 
(d) poor 



(b) good 
(e) zero 



(c) satisfactory 
(f) no answer 



His preparation for class is - 



(a) poor (b) satisfactory (c) good 

(d) less than satisfactory (e) exceUcnt (f) no answer 



5, The lecturer - 

(a) usually started and ended his class on time 

(b) sonetines started late or ended late 

(c) frequently started late or ended late 

(d) frequently started late and also ran over 

(e) often calls off class or doesn't show up 

(f) no answer 

6. The course as a whole was planned and organized - 



(a) extremely well 
(d) rather poorly 



(b) wel I fc) acceptably 

(e) extremely poorly (f) no answer 



7. In my opinion the inclusion in the course of lecture demonstrations » 
filnsy and visual aids was - 



(a) too Kuch 
(d) not enough 

The lecturer • 



(b) good 

(e) tco little 



(c) adequate 
(f] no answer 



ERIC 



(a) seldom wade the underlying physical ideas clear 

(b) never made the underlying physical ideas clear 

(c) usually made tlie underlying pUy-cical ideas clear 

(d) always nade the underlying physical ideas clear 
(f) no answer (i.e. leave blank) 

9. The lecturer's ability to make the subject interesting - 

(a) excellent (b) good (c) satisfactory (d] poor 

(e) zero (f) no answer 

10. The lecturer's apparent concern in teaching tliis class showed that he was - 

(a) exceedingly interested 

(b) usually quite interested 

(c) fairly interested 

(d) only occasionally interested* 

(e) bored 

(f) no answer 



!!• His attitude toward questions in class was - 

(a) very encouraging (b) usually encouraging 
(d) flippant (c) discouraging 



(c) adequately receptive 
(f) I have no opintoi; 
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3c, Part of Form 4 - Recitation Sections 

k.2 

^eetlons; 

1. With regard to this questionnaire: 

(a) You basically approve of such questionnaires, and hence you are responding 
to the vhole questionnaire. 

(b) You find such questionnaires an imposition, but In spite of this view 
you are responding to the whole questionnaire, ' 

(f) (l,e, leave blank) You disapprove of such questionnaires, and hence 

(PLEASE, after flUlng in the code ntenber) Intend to abstain canpletely, 

2. 1 evaluate the lecturer's Unovlci^.c of the subject cs boinj - 

(a) excellent (b) gooU (c) sati:;factoiy 

(d) less than satisfactor>- ^e) iucorpetcnt 
(f) no an sice r 

3. His ability to get the natcrial across to mc is - 

(a) excellent (b) good (c) satisfactory 

(d) poor (e) zero (f) no answer 



His preparation for class is « 

a) Poor (b) satisfactory (c) good 

d) less than satisfactory (e) excellent (f ) no answer 



5. The teacher - 

(a) usually started and ended his class on time 

(b) soractints started late or ended late 

(c) frequently started late or ended late 

(d) frequently started late and also ran over 

(e) often calls off class or doesn't show up 

(f) no ansKcr 

6. Hie teacher related the recitation class to the lectures » assignments and exans 

(a) extremely well (b) better than average 

(c) average (d) rather poorly 

(e) extremely poorly (f) no answer 

7. The teacher's ability to make the subject interesting was - 

(a) excellent (b) good (c) satisfactory (d) poor 

(e) zero (f) no answer 

Hie teacher* s apparent concern in teaching this class showed that he was - 

(a) usually quite interested 

(b) only occasionally interested 

(c) fairly interested 

(d) bored 

(e) exceedingly interested 

(f) no answer 

9. His attitude toward questions in class was - 

(a) very encouraging (b) usually encouraging 

(c) adequately receptive (d) flippant 

(e) discouraging (f) I have no opinion 

10. From my experience* the teacher's availability for questions outside of 
class hours was - 

(a) encouraging and generous of his t^me 

(b) usually available 

(c) * barely adequate * 

* (d) discouraging and usually unavailable 

(e) always unavailable 

(f) I have no opinion 
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11. 



12. 



The feeling between the teacher wd the students ki« 



(«) that of strong good will 

Cc) neutral 

(c) antagonistic 



(b) that of good will 
(d) negative 
(f) to answer 



In discussing homework or exam problems the teacher - 

"""f' """l^iyins physical ideas clear 
S s1?H^ "v''" underlying physical ideas cleir 
d! lll^ ' l^" physical ideas clear 

d never makes the underlying physical ideas clear 
(f) no answer (i.e. leave blank) 

IS. The teacher's attenntc rn , 

tenets to answer questions about the lecture was - 



(a) 
(b) 
(c) 
(d) 
(e) 



very clear and to the point 
Often clear and helpful 
usually satisfactory 
less than satisfactory 
generally unsuccessful 
no answer 



l'<- Class participation 



(«) 
(b) 
(c) 
(d) 
(e) 
(f) 



IS. 



effectively involved ever>one 
involved many people 
involved only a few people 
rarely involved anyone 
MS totally lacking 
no answer 



t'e'a^hin^r^orwo^fd P"^"" i" which this person was 



(«) 

(b) 
(c) 
(d) 
(e) 
(f) 



eagerly seek his section out 
take his section if convenient 
have no strong feelings 

•void hm like the plague 
no answer 



16. 



(a) 

(b) 
(c) 
(d) 
(e) 
Cf) 



is an outstanding teacher 
IS a good teacher 
has the potential of becomine a eood tMo».« 



17. 



His use of exa.^ies «,d illustrations was 



(a) 
(b) 
(c) 
W) 
(e) 
(f) 



excellent and effective 
good 

adequate and helpful 
less than satisfactory 
noticeably lacking 
no answer 



ERIC 





For 




the 




it; 




16. 




19. 




20. 




21. 




22. 




23. 


£r 


24. 


.1 


25. . 




26. 







inaudible 

writing is illegible 
speaks too fast 
speaks too slow 

poor organization on blackboard 
talks to the blackboard 
••hums" and 'liaws" 
distracting or nervous mannerisms 
any other bad habits? Mark (a) i 
yes and specify on the reverse 
Side of answer sheet 



if 



(a) 
(a) 
(a) 
(a) 
(a) 
(a) 
(a) 
(a) 



should be 
should be 
should be 
should be 
should be 
should be 
should be 
should be 



corrected 
corrected 
corrected 
corrected 
corrected 
corrected 
cor-^cted 
corrected 



(a) yes 
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3d. Part of Form 6 - "Unstructured" Self Contained Laboratory ^ ^ 
Course 
Quest lone : 

1« With regard to this questionnaire: 

(a) You basically approve of such queatlonnalres^ snd hence you are 
responding to the whole questionnaire. 

(b) You find such questionnaires an Imposition, but in spite of this 
view, you are responding tc the whole questionnaire. 

(f) (i.e. leave blank) You disapprove of such questionnaires, and 

hence (PLEASE, after filling In the code number) Intend to abstain 
completely. 



IMPORTANT — Vhen evaluating the laboratory teaching assistant* answer 
only the starred qaestlons. 



2. Kq'upment for e>*perijTent? was 

(a) adeq^^ite 
^b) iisadequate 

(r* ivj answer (i.e., leave blank) 
J. fquipment fov experiments was 

(a) adequately documented or explained 

(b) iriadequately iccumentcl or explained 
(f) no ar.swer fi.e,, led*'^ tjlank) 

♦U. The ips.-^^tc^or^s proparaticn 5or cla^s was 

(a) poor <b) satisfactory (c) good 

(d) less than satisfactory (e) c;. ct^j-xt jj*^ (f) no answer 

*S. Tbe instructor's knowledge of the physics in the c^q)erijnent was 

(a) excellent (b) good (c) adequate 

(d) weak (e) noticeably lacking (f) no answer 

*6. Iti the laboratory the instructor 

(a) was always available en-j willing to help 

(b) was adeouately hclpfr.l 

(c) was pres».::t, birt largely ignored t)»e laboratory 

(d) refiised to h-elp stucJents 

(e) freqi;er.tly left the laboratory 

(f ) no answer 

*7. When I had questions about the equipment, the instructor 

(a) was able to help me or direct me to someone with expert knowledge 

(b) showed less than adequate concern or interest 

(c) igr'Cr*»d ny picas for help 

(f) no answer (i.e., leave Dlank) 

*8. The instructor's apparent oonoem in teadiing this class shewed that he was 

(a) ustally quite, concerned 

(b) only occcisionally interested 
(c> fairly interested 

(d) bored 

(e) exceedingly interested 

(f ) IK) answer 



*9. His ability to get the material across to me is 

(a) excellent (b) good (c) satisfactory 

(d) poor (e) zero (f) no answer 

*10. The feeling between the jnotrucfor ard the students was 

(a) -that of ^;crwg goodwi.!! (b) that of goodwill (c) neutral 
(d) negative (e) antagonistic it) no answer 



His attitude tcwarti questions in class was 

(a) very encour>aging (b) usually encouraging (c) adequately receptive 
(d) flippant (e) disocuraging (f ) no answer ^ 

12. Grading and camcnts written on laboratory reports were 



(a) carefully done and helpful 

(b) adequately done 

(c) less than satisfactory / 

(d) ar^Jitrary and unfair 

(e) not done 

(f ) no answer 

*13. Ccfircnts and criticisms during oral laboratory reports were 

(a) carefully done and helpful 

(b) adequately done 

(c) less than satisfactory 

(d) arbitrary and unfair 

(e) not nade 

' f) no answer 

lU. Class presentations by the instructor were 

(a) worth the time they took 

(b) interesting, but the tiie required could have been better spent 
worJdng in the laboratory «^txer spent 

(c) of little value 

(f ) no answer 

15. The lecturer's 3bilit>- to n«ke class presentations interesting was 
(f) ^12^ ^""^ satisfactory (d) poor (e) zerx> 

16* The teacher's attenpts to answer questions about the lecture were 

(a) very clear and to the point 

(b) often clear and helpful 

(c) usually satisfactory 

(d) less than satisfactory 

(e) generally uisuccessful 

(f ) no answer 



*18. 



17. Class participation 

(a) effectively involved everycne 

(b) involved nany people 

(c) involved only a few people 

(d) rarely involved anyone 

(c) was totally ladcii^ 
(f ) no answer 

JLS^r;,:^w^2^^ ^^^^ P^^^- or^^rse ^ which this per.cn was 

(a) eagerly seek his section out 

(b) take his section if convenient 
ic} have no strmg feelings 

(d) avoid his section if easily possible 

(e) avoid hun lake the plague 

Ch the basis of this person's teaching in this course,. I think that he 

(a) is an outstanding teacher 

(b) is a good teacher 

(c) has the potential of becoming a good teacher 

^e) should not be retained as a teacher 

(f ) no answer 



*19. 



6. A 



A16 



For e&ch of the following possible bad habits, nark block (a) if and only 
iftt« instructor suffers from it, and it would be desirable for him tc 
correct it; othervise make no marie 



20. inaxKiible (a) 

21. writing is illegible (a) 

22. speaks too fast (a) 

23. speaks too slow (a) 
2U. poor organization on blackboard (a) 

25. talks to the blackboard (a) 

26. '^hums" and '^ws" (a) 

27. distracting or nervcjis ji^^erisms (a) 

28. any other bad habits? Mark (a) if (a) 
"yes," and- specify on the reverse 

side of the answer sheet 



should be corrected 
should be correcte^i 
should be corrected 
should be corr?cted 
should be correoit'd 
should be ccrrecteci 
should be corrected 
should be corrected 
yes 



Tar each of the follcwing positive qualities.^ marie block (a) if the 
instructor* has it to a high degree 



39. 



(a) yes 
(a) yes 



29. is articulate 

30. has excellent orgaTiization of the 

blackboard 

31. has a cheerful or pleasant disposition (a) yes 

32. speaks directly to the class (a) 

33. has a seTiSe of huncr when called for (a) 
3^. is very honest when he has nade a mistake 

or does not know the answer (a) 
35. avoids excessive rrathen-atical detail (a) 
35- Any other outstanding positive qualities? 
l^rixk block (a) if yes, and specify on the 
re^nyn^c side of answer s'ncet (a) 



yes 
yes 

yes 
yes 



yes 



37. Lficture presentations by students enrolled in the course: 

(a) should be oontiiiued 

(b) should not be required 

(c) should lie coritinued with a different format [naric block (c) 
and specify on rev;rse side of answer sheet 

vf ) no answer 

38. lecture presentations by students enix)lled in the course; 



(a) 
(b) 

(c) 
(f) 



wei>e wcr-th the ti/i^* Uiey tock 

vrere interesting, hut the time required could have been 
better spent wor'king in the laboratory 
were of little value 
no answer 



I perscTially found that the time I spent in preparing niy own talk 
and the experience of presenting it weixj 

(a) worthvMle 

(b) of mar^nal value 
(f ) no answei-^ 



W).* I found that the effort spent on this course was 



(a) excessive for the anount of credit given 

(b) mofre than that spent on other courses of equivalent credit 

(c) about the same as that spent on other courses of equivalent credit 

(d) less than that spent on other equivalent courses 

Due to my taking this course, my mastery of the subject matter and 
of the relevant skills aix3 nethods 



(a) 
(b) 
(c) 
(d) 
(e) 
(f) 



has 
has 
has 
has 
has 



iirproved greatly 
ijiprov^d sonewhat 
reireined constant 

decreased somewhat due to oonfusion created by the course 
decreased a grejt deal 



no answer 
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APPENDIX k. SAMPLE OUTFUTS SEEN BY INDIVIDUAL INSTRUCTORS 



JS? 4a, Sample PATS Output for an Individual Section 
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4b. 



Sample "Warnings" Riven for Participation Rates for Individual 
Sections 
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CUESTtONNAIRC NO. 2 

COURSE ifcl*^ SECTION OlQl 

instructor: krall 



STUoErnS WERE ENROUtrO IN THIS SECTION 
STUDENTS PARTICIPATING Cu^lPtFTEO ThE QUrsTIONNAIRE 
STUDENTS PARTICIPATING ChOSE TO ABSTAIN 
STUDENTS v^ERE NOT ACCOUNTED FOR 

PER CENT ( 122/ X7^) OF sTuOENTS ENROUUFD COMPLETED T.hE OUESTIOnNAIR^T 
• W i fi N I N C : LESS THAN 70 PER C^^NT OF ENROULEO STUOENTS COMPLETED THE Gu£STIONNAIR 

«*HYSICS FACUtTYf FALL I97i • TFAcHFnG SUhVtY RESULTS 
INDIVIDUAL SECTIONS 
• # 

C'JESTIONNAIRE NO. 2 

COURSE i«20«* SECTION niQt 



instructor: mcdonald# f 



17 students v»e«e enroi.lfo in this section 

0 students participating CO^'PLETED the dUrSTlONNAIRE 

1 STUDENTS participating ChOSE TO ABSTAIN 
Ito STUDENTS WERE NOT ACCOUi\l£0 FOR 

f ESTIMATE I 0 PER CENT ( 0/17) OF STuOtNTS ENROLLED COMPLETED THE QUESTIONNAIRE 

' THERE ARE NO RcSgtTS FOR THIS SECTION 



. <ESrlMATE> 179 
X22 
1 
56 

• (ESTIMATE^ 60 



I^ESTime) 
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2Jf 4c> Sample PATS Output for m Aggregate (Showing all 
^ Three Possible Wel^tings) 



ao 



y% 

UJ 

o: 

>• • 
u 
> 
ce 

iA 
Z 

r 
u 

< ui 

UJUI 
< 

r** < 
a 



u 
o 

M 

>• 
X 

a 



o 

Ul 



2 
2 
O 



I/I 
Id 
3 
9 



^1 
# I 
I 

U I 

f 

I 
I 
I 

5i 

I 
I 

• I 
I 

I 



< ! 

o'j. 

Ul I 

I 

K I 

o i 





• 




















Z 










# 






u 








Ul 




o 




*« 








(C 


(/I 


Ul 




z 








M 


2 


(A 












< 


o 














2 


M 














2 








o 








O 


o 














M 


Ul 


< 




* 
















Ul 








U) 




o 




Ul 








Ul 


>• 


Ul 














tt) 














o 




2 




o 










o 






Ul 








Ul 


Ul 


o 




K 








z 




u 




O 










z 






O 










o 


>• 




< 








o 


M 




Ul 




Ul 






Ul 


Ul 






in 


ec 








B 


Ul 


< 


z 








Ul 




a 


o 


o 


< 








1 


o 


Ul 




2 










flC 


(C 




2 








(/» 


a 


o 


u 


o 






o 


2 


z 


o 


Ul 


M 






u 


O 






10 










M 




o 




U) 






o 




X 




o 


W 






Ul 




u 


MUl 


Ul 




2 






00 




Z flO 


o 


O 






•J 


M 


Ul 


*- 






< 




o 


(C 






•J 


UJ 






QC 




Ul 


2 t/l 


u 


X 






2 


(/I 




¥4 


2 








Ul 


M 


< 








< 






o 




a 




o 






Ul 






UIUI 


Ul 


Ul 


o 


ct 




IL 




OUI 


(A 


1- 




o 


2 


o 






u< 


Ui 




w 


Ui 








Z 


•J 


Ul 




O 






o 




& 


(/I 


o 




Ul 


c 


2 10 






o 


Ul 


»• 


(/) 






2 


o 


Z 




(/I 










u 


u 






Ul 




u» 














Q 


(t Ul 


O 


o 


o 


o 


O 


o 


2 


< u 


Ui 


2 


k 


u 




o 


< 






M 




u 










•i 






< 




X 




u;u) 


O 


< 


< 










t* U) 


a 




& 






Ul 




< o 


2 






o 




z > 


o a 


UJ 


u 


U 


2 












M 


M 




in 




> 


a < 


Ui 






Ul 




Ul 




o 


(t 


ce 


z 


(K 






Ul 


ou. 


Ul 


< 


< 


Ul 




o 


u 


< o 




a. 


& 


'S 




2 


2 














o 


M 


in K 


1/) 




ut 


\A 








2 3 










2 






OO 


2 


2 


2 


2 


Ul 


• 






UJ 


Ul 


U2 


Ul 


U 


• 






c 


o 


O 










U 










X 






Ul 










Ul 






(/) 


Ut 


Ut 


Ut 


Ul 










CO 


tn 


o 


10 


•0 








*4 


#4 






9 



■:i 

I 



Ul 



Ul 
Ul 



i 



Ul 
Ul 



1 






• 


Ul 1 








H 1 








< 1 








O 1 








Ul 1 








a 1 








O 1 








O 1 








< 1 








Ul 1 








M 1 








Z 1 








H 1 








2 1 








M 1 




f 




O 1 








Ul 1 








SI 1 




< 




D 1 








^ 1 


z 






U 1 


o 






2 1 








M 1 


u 


Ul 


Ul 






u 


u 


Ul 1 








a 1 




Ul 


Ul 


< 1 


Ul 


>• 


>• 




2 


z 


z 


U) 1 
Ul 1 


M 


a 


a 


H 1 








< 1 








C 1 


o 


U) 


Ul 


Ul 1 




2 


2 


X 1 




O 


O 


O 1 




M 


M 


O 1 








< 1 




u 


u 


X 1 


2 


til 


Ul 


Ul 1 


O 


Ul 


Ul 


2 1 








O 1 




CM 


01 


M 1 


u 






H 1 


Ul 






U 1 


Ul 






Ul 1 








Ul 1 








O 1 
2 1 








M 1 
B 1 








O 1 




Ul 


Ul 


^ 1 








^ 1 




< 


< 


O 1 


Ui 


O 


o 


U. 1 


Ul 


Ul 


Ul 




fit 


X 


X 


Ul 1 


I> 


o 


o 


X 1 


o 


CO 


o 


»- 1 


u 


< 


< 



A22 



2? 

ON 
0 o 



(/I 
kJ 

or 

hi 
« 



WO 



•I 
u 

(A 

•« 

<A 
>• 
X 



I 
I 
I 

I 
I 

</) I 



u 



^\ 

I 

I 

LJ I 
H I 

< I 
O I 
U I 
Ct I 
O I 
O I 

< I 
I 
I 
I 

_ I 
I 

ft/I J 

H I 
J I 
3 I 
ftA I 
U I 

or I 
I 
I 
I 
I 

< I 

2 I 

°! 

1-4 I 
I- I 
I 

^ I 

3 I 
O I 



o 



o 

111 

CK 



z 
z 
o 



3 
O 



O .J 
ki ^ 

or < u; 
o D 
e c 

< kJ 

V 

2 Ci 

0 UIO 

I 

uo 
ft/)lJ 
o 

IJ w 

c: > 

3 O 

u cc 
o 

kJ 

1 H 
U ftA 



o« 

DO 
HO 

ftOO 
kJ 
X H O 

kJ 
It 



>* • 




m o 


u. 






D K 




kJ < 




Z U 








< o 


Ui 










O 




(A 




ftA < 




kJ 


z<o 




Ul D 




Q O 


H O 


3kJ 


Z « 




kJ H 




U O 




K kJ 




kJ ftA 


U X 


a 


< o 


o 


kJ w 


kJ kJ 


ki 


a o 


JCCD 














o z 




^ 1^ 




kJ 


< 


» kl 












X 




kJ O 




» z 








3 r- 




Z X 




O 


ftAU 


Ul M 




X Ul 




H at 


9Z 



• •••••••• ••• • 

94 ^ ^ c 



OOOOOOOGCOOOO 



&OOOff«CI OOO ^OOO OlO O K" I 



OOCCCOOOC'OOCOCC c 



K)h»h»io lO o o K>r-. loioioioooooooooooooooo' 



OlO 



•oriomh-iooioioiooioir. f^iof^oooooooor^coco^occcc^c 
^ * * * 



lO lO O lO «0 fO 
#4 #4 :f a <0 



10 lo meg fO *n wio 



O o 



^•^•lOoo^•^>lOo 



h-ioors'iOf^, looioh' oiooh.o»oiocoooo»o»n»n ^ 



OOfO CM ^ eo lO Cy W CM <M «-« CM CM <Vi * ^ O iT) fO 



ooooocooo ocf^c ^.o^•ooooo^oo^ooccooo^ 
««•«•••••••••••••••••••••••«••••••• 

v-iiOfOfOtf #4 <^ o «i o tr <r <r tr o 03 c tfj ^o^• * ^• 

*4 #4 ^ ^ 

OOOOQO0OOOOf^O000^>OOOO00OOOOOOO0000O 

iTi IT 4 Ai* ^ ti a ^ ^ a 



oooh>r*»ooco ooo 

CM*4 



ootf otf^•^>oo 



oooooooooooooc 



OONtfOtfONh-tfOOOOOlOOfOfOOOOOOOOOOOOOOOOO 

oa (O ^ «> ^ lOCMM « O CO 
*4tf>CM*4 «4jr*4lO<M *^«0 lOlO 



Kh>h>Otfh>9^it h>oh> tff* ooooooooo^»o 

******** 



00000990 



h«io mm o m o in o*o 



*4«o^mcMo r>iommomr>o 
M ^ if. k; #0 k. c ;r * k# 



^ ^f^oooooo ooe oooooooo 

to «) ^ •-imO^lO***^ #4 iT CM CM CM »4 



h>OOooooo 

*4 «0« #0 



O O O H> 



'J 

2 
0 

0 



7- 
J 
J 

0 

13 



«CM « 
fO CM 



•o in 
m (M 



OOK)OOOh>Oh> Oh> h> O h> O O h> Oh- •OOh-OiOiOh-h'OOOOCf*!^ 
10 O ^ 



«o «c o o o o «0 

<M CM CM IM ^ 



«OlOO^Or)lO«0^00000<0^9^0 



ooiOOOiOh>h>r*>h>oooooo 



CM — 



» o 10 o o o o ^ 00 
010 «o 

CM w 



00l00l0h>h>000900 

10 m «o «o 



0000000000 



0000000000 



OOlOOOlOh>h>lO lOO h» O lO h» lOlOlOlOOOOOOOOOOOOOOOO o 



I000l0<0<d*0 ^iOiO 
#4 «OCM *4 IMIO ^ CM 



fO « 10 «o m #0 

#4 10 #4 10 10 



lOlOOlOh>lCOlOiO lOOlOlOf^lOf^OOOOOl 



>oooooooooo 



10 10 OlO ^10 

^ #4 ^ ^ ^ 10 



I0«0l00l0«0^f0<0 

•OlO mcM 10 m CM 10 



o o 



h.t*»rtooh»h«iooh»iooh»toioiooioh»h.oiooKo<^io©ecooioiOi'' 
owD^iootO*^©^*nioiooio^^ 10 ^ 

COCIO CM * CO 10 CM CM CM CM »0 »4 10 CM CM 



lOlO e oomioio 
#4«-< CM ^tfi^mio 



•4CMr)^(niOh* coo^ o*^cmio crm^or-c o»o .^cmio ^in^h> ccc^o-^^cMOtfrn 

*^ «^ .-f « «^ w ^^^^^W*^*<CMCMCMCMCM<VCMCMCMCMIOIOIOIO*OIO 



APPENDIX 5. THE COMMENT SHEET USED ( JNSUCCESSFULLY) IN SUMMER 1971 
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Course Instructo r 1 — j | | | | | 

lect/recit/lab; instructor/TA I III} 

(delete as appropriate) cqdE NUMBER 

University of Maryland 

Physics Department 
Teaching Questionnaire 

Comment Sheet 

ANY COMMENTS, ETC., MUST BE PUT ON THIS SHEET. If you make any comments, etc., 
\LSO fill in the identifying information. Comment sheets with responses should be 
tianded in separately from the standard answer sheets. 

On Course 



Section 

Summer 1971 



On Teacher 



On Texts 



On Exams 



On Homework 



On Questionnaire 



Other 




Continue on second comment sheet if necessary • 



APPENDIX 6. AN APPROXIMATE BUDGET FOR PATS 



MEMO 



TO: Howard Laster 
FROM: C. Kacser Prepared Dec. *71 
SUBJECT: Physics Questionnaire Budgets 



(We exclude Astronomy costs) 

1. Fall 1970 Short Form 

This was a one page questionnaire, the same for all situations. Stu- 
dents marked squares on the form, and the responses were then keypunched by 
hand by the C.S.C. Programming was done by Katz and Fram, two high energy 
graduate students, and computing was paid for by the Department. 2818 re- 
sponses were processed . 

Keypunching $491 (dept. to C.S.C.) 

Computing (debugging and "$561 (dept. to C.S.C.) 

final run)- 

Katz and Fram programming $200 (dept. to individual) 

Printing of Questionnaire, $20 
etc. (estimate) 



Total Cost as paid by Dept. $1072 

Cost /response .38c 

2. Spring 1971 Six Version Long Form 

This consisted of typically a five page, 60 uestion form, answers to 
be marked in pencil on a special standard answer sheet. These sheets were 
then optically read and converted into IBM cards ("digitek** process). 
Cards were arranged by hand with "header" cards, keypunched by a student 
worker, and then initially processed using the Cyclotron IBM 360 (within 
the Department) and using a program developed by Tom Whitp of the Cyclotron 
group (i.e. again within the department). The individual section output 
so generated was then used as input for a further "aggregating" program. 
This program was developed by Fram and Trewett , two high energy graduate 
students, and run on the C.S.C. 1108. About 2825 responses were processed. 
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Digitek (4c/response) 
1108 Computing 

paid by dept* 

supported by C.S.C. 

other budget support 

(Siibtotal) 

1108 Programming 

(Fram and Trewett) 

1108 Data preparation 

Cyclotron programoing 

Cyclotron computing 

Total Data Processing Costs 
Pencils (estimate 4c x 6000) 
Answer sheets ($5.63/box of 500) 
Printing/ collating/ stapling 
Total Cost as paid by dept. 
Cost /response 

NOTE - This includes non- 
recurring development costs 



nominal 
cost 

$113 



$353 
$2511 
^.$1600 
$4464 

$830 

^.$50 
10-15hrs. 
lOhrs. 

$4627 
'v.$240 
'x.$30 
$415 



actual cost 
paid by Dept. 

$113 



$353 



$1266 
$240 
$30 
$415 



$1951 



.69 



(to C.S.C.) 



(to C.S.C.) 



$353 

$800 (to individual) 



(in house) 



Intangible Costs 

Overall programming and processing supervision by 
Claude Kacser 

Administration, preparation of packets and col- 
lection of output by Dr. Griggs 

Sharpening pencils (secretary) 

Devising questionnaires (faculty) 

(Homyaky Kacser, et.al., incl. Jearl Walker) 

Proof reading, etc. , etc. (C. Kacser) 



3-5 weeks? 
10 dssys 

2 Secretary /days 

4 Faculty/days 
2 days 



3. Fall 1971 Six Version Revised Long Form 

Estimates * (Note that Astronomy is excluded) 



This is essentially the same format as Spring '71. The programming 
need not be redone so that this might correspond to a "steady state." 
About 8^000 physics forms are being solicited^ assume 7^000 responses. 



Digitek (4(/response) 

1108 Co]q>uting 

supported by C.S.C. 

Cyclotron computing 

Total Data Processing costs 

Pencils (replacements » etc. 
4C X 6000) 

Answer sheets ('V'l.l^ x response) 

Printing/collating/stapling 
(Physics only) 

Total cost as paid by dept. 

Cost/response 



actual cost 
paid by Dept. 

$280 (to C.S.C.) 



$280 
$240 

$80 $80 
$540 $540 (in house) 

$Ii40 

.16 c 



nominal 
cost 

$280 

$800 
6hrs 
$1080 
$240 



Intangible *Josts 



Overall processing supervision by G. Snow and 6 days? 

C. Kacser 

Data preparation^ keypunching, etc. (Krs. Dobbins) 3 days? 

Sharpening pencils » preparing packets » etc. 6 Secretary/ days 

Administration and collection of output by 10*>15 days 
Dr. Griggs 



Notes: 1. The pencil costs could be perhaps reduced in future by 
sawing present stock in half. But sharpening is then a problem. Also if 
8^000 are given out, maybe 4,000 come back. In future we should only need 
4,000 X 4c $160 new pencils /semester? 



2. On present experience of administrlng the Fall '71 question- 
naire, we seem to be getting 3 out of 4 pencils returned. Also we are 
asking that the questionnaires be returned, and hence we should have nearly 

ZTt/j'''^ II' II ''f ^^'^'^^ P^^"'^^"^ costs [provided no changes 

are made.] Thus the cost for Spr. '72 might well be less than 18c/response. 

3. Digitek rates will go up from 4c to lOc on July 1, 1972. 

?oiLl"?r ??f ^"^^ "^^ht be persuaded to\e;p cos^s 

lower.) It will add 6c per response , 

4. Will the C.S.C. give us continuing free support for 1108 
computing in future for an "operating" academic expense? It has already 
made clear that it feels it should not. 

5. One intangible cost, not adequately estimated or Included, 
is the preparation of control cards for AGGFORM. This is "easiest" if 
done by a fairly experienced programming assistant or graduate student, 
and might then take 2-4 days? 



